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Preface 


The  purpose  of  this  study  was  to  validate  the  decision 
making  process  TAF  uses  to  prioritize  their  potential  R&M 
improvement  modifications  and  to  suggest  other  possible 
approaches  to  solve  their  prioritization  problem. 

In  performing  the  research  and  writing  this  thesis  I 
have  received  a  great  deal  of  help  from  others.  I  am  deeply 
indebted  to  my  faculty  advisor,  Capt  Joseph  Tatman,  for  his 
enthusiasm  and  his  words  of  encouragement.  I  would  also 
like  to  thank  my  thesis  reader,  Lt  Col  Thomas  Schuppe,  for 
his  thoughtful  insights. 


Katherine  H.  Darko 


Table  slL  Contents 


Page 

11 


Preface . 

List  of  Figures 
List  of  Tables. 


v 

vl 


Abstract .  vll 

I.  Introduction .  1 

Specific  Problem  Statement .  1 

Motivation  for  Managers .  2 

Subobjectives  of  the  Research .  3 

II.  Background .  5 

What  TAF  Heeds  and  Why .  5 

TAF's  Process  to  Develop  the  List .  8 

III.  Methodology  and  the  TAF  Decision  Making  Process..  16 

Steps  Done  in  the  Validation .  16 

Steps  for  Information  Gathering .  16 

General  Steps .  18 

Making  a  Decision .  19 

AHP  Applicability . 23 

IV.  Solution  within  the  AHP  Framework... .  27 

Analysis  of  TAF's  Use  of  the  AHP .  27 

Improvement  to  TAF's  Tree .  44 

Further  Improvement  to  NEWTREE .  55 

Ideal  R&M  2000  Tree .  56 

V.  Problems  within  the  AHP .  63 

Dependency .  63 

Uncertainty .  68 

Comparison  Scale .  72 

VI.  New  Techniques .  78 

Decision  Analysis .  79 

PROMBTHEE .  83 


ill 


List  o±  Figured. 


Figure  Page 

1.  The  Current  TAF  Tree  (TAFTREE) .  11 

2.  Teeter-totter .  14 

3.  The  Current  TAF  Tree  (TAFTREE) .  29 

4.  Current  Data  Worksheet .  39 

5.  New  Worksheet  for  the  Data .  40 

6.  Improved  Tree  (NEWTREE) . 45 

7.  Tree  for  Further  Improvement 

of  TAF's  Tree  (NEW2TREE) .  57 

6.  R4M  2000  Tree .  59 

71 


9.  Decision  Tree  for  Example 
10.  Decision  Diagram . 


80 


V  * 


Table 


Li&t  ai.  Tables 


Page 


1.  Data  Used  to  Show  Improvement .  52 

2.  R&M  2000  Goals  and  Criteria .  60 

3.  Prioritizations  Yielded  by  the  AHP  and  DA .  72 


vl 


VALIDATION  OF  THE  TACTICAL  AIR  FORCE'S 
DECISION  MAKING  PROCESS 
TO  PRIORITIZE  MODIFICATIONS  USING 
THE  ANALYTIC  HIERARCHY  PROCESS 

Introduction 

Annually,  the  Tactical  Air  Force  (TAF)  needs  to  develop 
a  prioritized  list  of  Its  reliability  and  maintainability 
modifications  (mods)  to  aid  the  decision  makers  who  allocate 
funds  to  the  modification  programs.  To  develop  this  list, 
TAF  uses  a  decision  making  process  called  the  analytic 
hierarchy  process  (AHP).  Since  the  prioritized  list  is  so 
Important,  TAF  Is  Interested  In  validating  their  use  of  the 
AHP.  The  validation  will  need  to  consider  three  areas:  (1) 
how  well  TAF  satisfies  the  criteria  necessary  to  make  a  good 
decision,  (2)  If  TAF  Is  correctly  using  the  AHP,  and  (3) 
shortcomings  that  exist  within  the  AHP  that  may  pose 
problems  for  TAF. 

specific  p.Lafclftm  Statement 

TAF  has  not  yet  validated  the  process  with  which  their 
prioritized  list  of  mods  Is  developed.  The  purpose  of  this 
research  is  to  validate  TAF’s  decision  making  process  that 
generates  a  prioritized  list  of  reliability  and 
maintainability  modifications  and  to  suggest  other  possible 
approaches  to  solve  the  prioritization  problem.  The 
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validation  will  be  done  in  three  parts.  The  first  part  will 
consider  the  criteria  one  should  follow  when  making  a 
decision  and  then  demonstrate  how  well  TAF  succeeds  in 
fulfilling  these  criteria.  Also  discussed  are  the  reasons 
for  using  the  AHP  approach  to  solve  a  problem.  This  is 
followed  by  an  analysis  of  the  appropriateness  of  TAF 
solving  their  specific  problem  using  the  AHP.  Second,  TAF's 
use  of  the  AHP  will  be  scrutinized  to  determine  areas  where 
they  may  not  be  performing  optimally.  Third,  some 
shortcomings  that  are  contained  within  the  AHr  will  be 
presented  since  they  may  pose  hidden  problems  to  TAF.  The 
second  part  of  the  research  will  propose  alternative  methods 
to  solve  TAF's  prioritization  problem.  These  methods  will 
improve  on  or  eliminate  some  of  the  identified  problems  with 
TAF's  AHP  approach. 

tteUyatiao  £ar  Managers 

This  paper  may  assist  decision  makers,  who  are  faced 
with  a  complex  decision  to  make,  in  three  areas.  First, 
ideas  on  some  criteria  that  should  be  considered  and  met 
when  trying  to  make  a  decision  involving  the  ranking  of 
various  alternatives  will  be  presented.  The  criteria  will 
help  him  to  determine  if  the  decision  process  used  was 
satisfactory  by  supplying  him  with  a  set  of  standards  to 
compare  the  process  against.  Second,  the  AHP  will  be 
explained  in  enough  detail  to  enable  the  decision  maker  to 
decide  if  this  technique  would  apply  to  solving  his 
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particular  decision.  If  the  AHP  seems  an  appropriate 
method,  the  decision  maker  will  be  introduced  to  ideas  on 
how  best  to  accomplish  the  technique,  as  well  as  ideas  on 
some  problems  he  may  encounter.  Third,  this  paper  will 
present  alternative  methods  to  solve  complex  decision 
problems  that  may  better  apply  to  the  decision  maker's 
specific  problem. 

auhobifiCtlyes  oJL  the  Research 

1.  Validation  of  the  decision  process 

A.  Identify  the  criteria  that  need  to  be  considered 
when  making  a  decision  such  as  TAF's  decision 

B.  Analyze  TAF's  fulfillment  of  these  decision  criteria 

2.  Validation  of  the  applicability  of  the  AHP 

A.  Identify  reasons  to  use  the  AHP  when  making  a 
decision 

B.  Analyze  the  existence  of  these  reasons  in  TAF's 
decision  problem 

3.  Validation  of  the  application  of  the  AHP 

A.  Identify  several  major  areas  where  TAF  may  encounter 
problems  in  their  use  of  the  AHP 

B.  Analyze  TAF's  use  of  the  AHP  considering  these 
problem  areas 

C.  Suggest  ways  TAF  may  overcome  these  problems 
(restructure  their  existing  tree  and  develop  a  new  R&M  2000 


4.  New  methods  to  do  the  prioritization  of  the  mods 

A.  Determine  what  other  methods  exist  for  prioritizing 
TAF's  alternatives 

B.  Determine  the  relevancy  of  these  methods 

C.  Determine  if  these  methods  would  improve  the 
prioritized  list  obtained  through  the  AHP 


This  chapter  is  divided  into  two  sections.  The  first 
section  provides  the  background  necessary  to  understand  why 
the  Tactical  Air  Force  needs  a  prioritized  list  of  roods. 

The  second  section  explains  how  the  Tactical  Air  Force 
develops  the  list  using  the  analytic  hierarchy  process. 

What  IAE  Meeds  and.  Why 

The  United  States  Air  Force  meets  a  significant  portion 
of  its  mission  requirements  through  the  deployment  of  new 
weapon  systems  and  the  modification  of  existing  weapon 
systems.  "Modifications  (mods)  add  new  capabilities, 
correct  deficiencies,  improve  reliability  and 
maintainability,  and  extend  the  life  of  existing  weapon 
systems  (6:1)".  This  paper  will  focus  on  Class  IVB  mods 
which  are  those  that  "change  a  configuration  item  for 
reasons  of  material  deficiencies  or  to  improve  reliability 
and  maintainability  ( R&M )  (6:1)".  The  acquisition  of  Class 
IV  mods  is  currently  being  funded  at  a  level  of  over  one 
billion  dollars  a  year  (6:1).  Due  to  limited  funds,  the 
Class  IV  mods  must  be  prioritized  on  an  Air  Force-wide 
basis.  This  prioritization  is  accomplished  by  Headquarters 
Air  Force  Logistics  Command  using  the  H040  algorithm.  Using 
the  data  supplied  by  the  system  program  managers  working  in 
conjunction  with  the  operating  commands,  this  algorithm 
produces  an  Integrated  Air  Force  Logistics  Command  Class  IV 


mod  ranking.  This  listing  is  then  submitted  to  HQ  USAF/LE- 
RD  at  the  Pentagon  in  support  of  the  Program  Objective 
Memorandum  and  the  Budget  Estimate  Submission  (6:2). 

The  Tactical  Air  Force,  consisting  of  the  Tactical  Air 
Command,  United  States  Air  Forces  in  Europe,  Pacific  Air 
Forces,  Alaskan  Air  Command,  the  National  Guard  Bureau,  and 
the  Air  Force  reserves,  is  one  of  the  operating  commands  who 
submit  a  prioritized  list  of  their  mods  to  the  Air  Force 
Logistics  Command.  Developing  this  prioritized  list  is  a 
difficult  task.  The  Tactical  Air  Force  could  not  use  the 
HO40  algorithm  to  do  the  prioritization  since  the  HO40 
algorithm  was  developed  in  1979  and  is  primarily  based  on 
the  mod  status  (i.e.  its  feasibility  or  budget  execution 
(6:5))  rather  than  the  system  program  managers'  priority, 
mission  need,  and  reliability  and  maintainability 
improvement  (6:iii). 

The  Air  Force  renewed  its  commitment  to  reliability  and 
maintainability  (R&M)  in  1985  by  "placing  R&M  coequal  with 
cost,  schedule,  and  performance  for  research,  development, 
acquisition,  and  modification  of  future  and  existing  weapon 
systems  (24:1-2)".  This  renewed  commitment  was  embodied  in 
the  United  States  Air  Force  R&M  2000  Plan  (24:v).  Since  the 
HO40  algorithm  was  developed  six  years  before  the  R&M  2000 
push  it  does  not  consider  the  goals  laid  out  by  the  R&M  2000 
plan.  However,  the  Tactical  Air  Force  is  required  to  follow 
the  plan. 


Thus,  the  Tactical  Air  Force  needed  to  £lnd  a  way  to 
prioritize  Its  Class  IVB  mods  that  took  cost,  mission,  and 
Improved  R&M  Into  consideration.  Since  it  is  difficult  to 
compare  such  diverse  characteristics  of  the  mods,  the 
Tactical  Air  Force  realized  they  needed  a  process  that  could 
handle  the  diversity.  In  a  similar  situation,  the  Tactical 
Air  Command  had  used  the  Planning  and  Programming  Priority 
Project  (P4)  model  to  rank  program  objective  memoranda.  In 
this  case  the  decision  makers  had  to  rank  very  diverse 
projects  such  as  installing  a  new  F-15  wing  or  a  new 
gymnasium  (22).  The  P4  model  was  chosen  in  this  case 
because  it  used  the  analytic  hierarchy  process  which  is 
"suitable  for  solving  complicated  and  elusive  decision 
problems  (27:96)".  Since  the  P4  model  yielded  such 
outstanding  results  in  this  application  to  the  program 
objective  memoranda,  the  Tactical  Air  Force  decided  to  use 
it  to  rank  its  modifications  (22). 

The  prioritized  list  that  is  derived  from  the  output 
from  the  P4  model  is  sent  to  AFLC  LOC/CC  and  AFLC/CC  who  are 
in  charge  of  the  HO40  model.  The  list  is  also  forwarded 
directly  to  USAF/  LE-RD  so  that  the  people  there  will  know 
the  Tactical  Air  Force's  priorities  on  the  mods  before  the 
mods  are  put  through  the  HO40  algorithm  (5).  The  list  is 
also  published  annually  in  the  Tactical  Forces  Multicommand 
R&M  2000  Plan,  Volume  II  (24:vl).  Due  to  the  high 
visibility  of  the  list,  the  Tactical  Air  Force  needs  a 
validation  and  possible  Improvement  of  the  process  they  use 


to  do  the  pr locitization  of  the  mods.  In  other  words,  they 
need  their  use  of  the  analytic  hierarchy  process  validated. 

TAT  'a  eraceaa.  ta  Q&y&lap  tiia  List 

Remember  that  the  model  used  by  the  Tactical  Air  Force 

to  develop  the  prioritized  list  of  their  mods  is  based  on 

the  analytic  hierarchy  process.  The  theory  of  the  analytic 

hierarchy  process  had  its  beginnings  in  1971  (19:iv).  Saaty 

believed  then  that  a  process  could  be  developed  to  aid 

decision  makers  in  reducing  "formidably  intricate  systems  to 

a  sequence  of  pairwise  comparisons  of  properly  identified 

components  (19:4)".  For  example,  Saaty  considers  the 

familiar  complex  problem  of  comparing  apples  to  oranges. 

An  apple  and  an  orange  may  have  many  characteristics  in 
common:  size,  shape,  taste,  aroma,  color,  seediness, 
juiciness,  and  so  on.  In  addition,  the  strength  of  our 
preference  for  these  characteristics  may  vary.  We  may 
be  indifferent  to  the  size  and  color  but  may  have  a 
strong  preference  for  taste  which  again  may  vary  with 
the  time  of  day  (19:xll. 

With  the  goal  of  being  able  to  deal  with  problems  similar  to 
this  one,  Saaty  developed  the  analytic  hierarchy  process. 

The  analytic  hierarchy  process  solves  decision  problems 
in  four  steps.  First,  a  decision  hierarchy  that  structures 
the  problem  is  set  up  by  the  decision  analyst  (23:98).  This 
structure  places  the  decision  objective  at  the  top, 
attributes  that  contribute  to  the  quality  of  the  decision  in 
the  middle,  these  will  be  called  decision  elements,  and 
decision  alternatives  at  the  bottom  (27:96-97).  For 
example,  in  the  apple  and  orange  comparison,  the  top  level 
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will  be  the  objective  of  choosing  a  piece  of  fruit  to  eat. 
The  characteristics  of  the  fruit  would  occupy  the  middle 
levels,  and  the  decision  alternatives  of  types  of  fruit  to 
eat  would  occupy  the  lowest  level.  The  second  step  In  the 
analytic  hierarchy  process  Involves  the  collection  of  data 
by  "pairwise  comparison  of  decision  elements  (27:96)".  In 
our  example,  data  could  be  collected  by  comparing  the  taste, 
color,  and  juiciness  of  apples  to  oranges.  The  third  step 
In  the  process  Is  a  solution  step  that  establishes 
priorities  among  elements  of  the  hierarchy  (23:98).  This 
step  would  yield  relative  weights  for  the  importance  of 
taste,  color,  and  juiciness  in  the  apple  and  orange 
comparison.  The  fourth  step  aggregates  the  relative  weights 
from  the  third  step  and  produces  a  "vector  of  composite 
weights  which  serve  as  ratings  of  decision  alternatives... 
in  achieving  the  most  general  objective  of  the  problem 
(27:99)".  In  other  words,  it  ranks  the  decision 
alternatives  from  the  best  to  the  worst.  With  this 
introduction  to  the  analytic  hierarchy  process,  the  Tactical 
Air  Force's  application  of  its  four  steps  will  now  be 
discussed . 

To  accomplish  the  first  step  of  the  analytic  hierarchy 
process  the  Tactical  Air  Force  had  to  do  two  things. 

First,  it  had  to  structure  its  problem  by  setting  up  the 
decision  hierarchy  and  second,  it  had  to  identify  all  the 
potential  mods.  The  decision  hierarchy  was  set  up  by 
Identifying  the  decision  objective  as  getting  the  most  r&m 


improvement  benefit  while  minimizing  the  cost.  The  decision 


elements  were  identified  as  reliability  improvement, 
maintainability  improvement,  inventory  size,  mod  cost,  and 
remaining  life  cycle  (see  Figure  1).  The  decision 
alternatives  (although  not  shown  in  the  figure)  are  the 
proposed  mods.  The  entire  set  of  proposed  mods  are  located 
directly  beneath  each  of  the  five  final  decision  elements. 
For  example,  if  there  are  three  proposed  mods  then  the  mod 
cost  node  would  look  like  the  following. 


To  identify  all  the  potential  mods,  the  Tactical  Air 
Force  needs  to  contact  the  using  commands  of  the  aircraft, 
specifically  the  system  program  managers  for  the  various 
Tactical  Air  Force  aircraft  (5).  The  system  program 
managers  at  the  Air  Force  Logistics  Centers  receive  all  the 
proposed  mods  for  their  aircraft  by  two  means.  First,  the 
field  units  will  identify  mods  and  submit  them  to  their 
Tactical  Air  Command  functional  managers  who  forward  the 
mods  to  the  system  program  manager.  Second,  the  staff 
agencies  of  the  major  commands  will  see  pressure  points  and 
contact  the  system  program  manager  directly  with  potential 
mods  (5).  The  Tactical  Air  Force  collects  these  mods  every 
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November  and  pulls  out  the  most  important  ones.  These  mods 
are  the  ones  that  ultimately  appear  as  the  decision 
alternatives  at  the  bottom  level  of  the  decision  hierarchy. 

The  second  step  of  the  analytic  hierarchy'  process 
requires  first  that  relevant  data  be  collected  on  the 
various  decision  alternatives  and  second  that  a  pairwise 
comparison  of  the  decision  elements  be  performed.  The  data 
that  the  Tactical  Air  Force  uses  is  collected  from  several 
sources.  First,  data  on  the  reliability  and  maintainability 
improvements  come  from  the  Maintenance  and  Operational  Data 
Access  System  (MODAS),  which  is  a  "central,  on-line, 
interactive  data  storage  and  processing  of  maintenance  and 
operational  data  on  selected  USAF  weapon  systems  (12:2-1)", 
under  the  Air  Force  Logistics  Command.  The  data  for  the 
Inventory  size  and  the  mod  cost  come  from  the  Air  Logistics 
Centers  of  the  specific  aircraft  that  the  mod  will  be 
performed  on.  Data  on  the  remaining  life  cycle  of  the 
aircraft  on  which  the  mod  will  be  done  is  a  figure  drawn 
from  knowledge  of  the  aircraft  and  an  educated  guess  of  the 
year  its  replacement  will  occur.  All  of  the  data  collected 
is  entered  on  a  worksheet;  one  worksheet  for  each  proposed 
mod.  These  worksheets  are  filled  out  by  the  program 
managers  residing  at  the  Air  Logistics  Centers  where  the 
various  aircraft  systems  are  located  (5). 

The  Tactical  Air  Force  takes  this  data  and  does  a 
pairwise  comparison  of  each  decision  element  for  the 
proposed  mods,  which  are  the  decision  alternatives.  To 


accomplish  this  step,  a  panel  of  about  ten  people  with 
experience  in  the  various  Tactical  Air  Force  aircraft  are 
chosen  from  the  Headquarters  Tactical  Air  Command  aircraft 
functional  managers/  product  improvement  personnel  from  the 
Tactical  Air  Command/  and  Headquarters  Tactical  Air  Command 
SMO-R&M  personnel  (5).  The  panel  meets  for  approximately 
two  full  days  and  makes  the  comparisons. 

For  example,  assume  there  were  three  mods;  A,  B,  and  C 
(decision  alternatives),  and  the  decision  elements  the 
Tactical  Air  Force  has  identified  in  their  hierarchy  tree. 
The  panel  would  compare  mod  A  to  mod  B  in  terms  of 
reliability  improvement  and  determine  which  one  was  better 
and  by  how  much.  In  this  case,  mod  A  is  strongly  preferred 
to  rood  B  in  reliability  improvement  (see  Figure  2).  Then  mod 
A  would  again  be  compared  to  mod  B  in  terms  of 
maintainability  improvement.  These  comparisons  would 
continue  for  all  five  decision  elements.  Then  mod  A  would 
be  compared  to  mod  C,  and  then  mod  B  to  mod  C.  The  new  data 
generated  by  these  pairwise  comparisons  are  now  available 
for  use  in  the  fourth  step  of  the  analytic  hierarchy 
process . 

The  third  step  of  the  analytic  hierarchy  process 
establishes  priorities  among  the  decision  elements  of  the 
hierarchy.  On  the  Tactical  Air  Force  tree,  for  example,  the 
relative  Importance  of  the  overall  benefit  to  the  overall 
cost  has  been  set  as  .7  to  .3.  In  other  words,  benefit 
contributes  70%  to  the  goal  of  highest  R&M  benefit  for  the 
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Figure  2.  Teeter-totter 


minimum  cost  while  cost  only  contributes  30%  to  this  goal 


The  Tactical  Air  Force  established  these  weights  with 


knowledge  of  what  their  organization  ultimately  felt  was 


important . 


The  fourth  step  of  the  analytic  hierarchy  process 


aggregates  all  the  weights  to  yield  a  value  for  each  mod. 


For  example,  if  mod  A  had  a  weight  of  .2  for  reliability 


Improvement,  .2  for  maintainability  improvement,  .6  for 


Inventory  size,  .6  for  mod  cost,  and  .1  for  remaining  life 


by  summing  up  all  the  values  at  each  end  node  multiplied  up 
through  the  tree.  The  way  the  Tactical  Air  Force 
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Ill .  Methodology  and.  the.  TAF  Baclalon  Making  Process 

Steps  Done  In  the  Validation 

This  and  the  next  two  chapters  Investigate  the  decision 
process  TAP  uses  to  arrive  at  a  prioritized  list  of  R&M 
mods.  This  chapter  looks  at  the  general  decision  and 
discusses  several  problems  TAF  might  be  facing.  Then,  given 
that  the  decision  is  structured  and  uses  the  AHP  to  solve 
the  problem.  Chapter  4  presents  and  critiques  TAF's  use  of 
the  AHP.  Last,  given  that  the  the  AHP  is  properly  applied, 
hidden  problems  within  the  process  itself  are  explored  in 
Chapter  5. 

This  chapter  will  begin  by  briefly  presenting  the  steps 
taken  to  acquire  the  information,  data,  and  understanding 
required  to  satisfy  TAF's  need  for  a  validation  of  their 
decision  process.  This  Is  followed  by  a  very  general  view 
of  what  steps  need  to  be  taken  for  any  decision  to  ensure 
that  the  decision  made  is  a  good  one.  A  discussion  on  how 
well  TAF  accomplishes  these  steps  will  follow.  Then  a 
section  on  the  specific  criteria  TAF  needs  to  meet  for  their 
decision  to  be  satisfactory  is  presented.  The  last  section 
of  the  chapter  discusses  the  applicability  of  the  AHP  to 
TAF's  decision  problem. 

steps  lax.  Information  gathering 

The  steps  taken  to  gather  the  information  necessary  to 


accurately  validate  TAF's  decision  making  process  for  the 


prioritization  o£  mods  can  be  divided  into  the  four  main 
categories  of  discussing  ideas  with  the  analyst  who 
requested  the  validation,  discussing  topics  with  experts  in 
various  fields  related  to  the  research,  conducting  an 
extensive  literature  review  on  many  related  topics,  and 
learning  how  to  use  a  software  package  called  Expert  Choice 
(7).  Each  category  will  now  be  discussed  in  detail. 

A  trip  was  taken  to  Langley  AFB,  Virginia  in  June  of 
1987  when  the  research  was  just  beginning.  Two  days  were 
spent  discussing  the  process  TAF  uses  in  prioritizing  its 
mods  and  their  need  for  a  validation  of  their  process. 
Captain  Wallace  Collins  of  TAC  SMO-R&M  was  the  analyst 
contacted.  Appropriate  reports,  computer  runs,  and 
correspondence  were  obtained  for  background  information. 
Questions  that  were  prepared  prior  to  the  meeting  were 
answered  in  sufficient  detail  to  permit  progress  in  the 
research  effort.  A  visit  was  made  to  Lieutenant  Richard 
Schooff  of  TAC/XP-JSG,  also  located  at  Langley  AFB.  From 
him  many  insights  into  the  workings  of  the  model  used  by  TAF 
in  solving  their  problem  were  gained. 

Following  the  trip  to  Langley,  a  need  for  detailed 
information  about  data,  R&M  2000,  and  various  other  topics 
was  needed.  At  this  point  experts  in  the  various  fields 
were  contacted.  Some  names  were  provided  by  Captain  Collins 
and  some  by  the  thesis  committee.  This  acquisition  of  the 
necessary  information  was  an  Iterative  process  in  certain 
situations  due  to  the  complexity  of  the  topics. 


The  literature  review  was  done  from  the  beginning  to 
the  end  of  the  thesis.  As  more  areas  were  uncovered  and 
more  questions  arose,  the  need  for  articles  by  experts  In 
the  field  were  needed.  The  literature  search  was  focused 
primarily  on  the  AHP,  Its  applications,  and  its 
shortcomings.  Many  articles  were  also  found  dealing  with 
the  subject  of  decision  making,  specifically  multiple- 
criteria  decision  making. 

The  software  package  Expert  Choice  which  is  used  with 
the  micro-computer  was  invaluable.  It  was  used  to  solve 
sample  decision  problems  using  the  AHP.  This  package  was 
used  for  all  the  examples  presented  in  this  paper.  It 
allowed  for  quick  changes  in  the  decision  tree  structure  and 
weights  so  that  it  was  usually  an  easy  task  to  find  a  case 
to  prove  a  point  trying  to  be  made. 

qg petal  Steps 

In  any  decision  there  are  certain  specific  steps  that 
must  be  taken  to  ensure  that  the  final  decision  made  is  a 
good  one.  First,  the  problem  or  decision  must  be  clearly 
defined.  This  involves  specifying  a  goal.  Second,  one  must 
state  how  the  decision  will  be  evaluated.  This  involves, 
for  example,  identifying  performance  measures.  Third,  the 
alternatives  that  are  available  with  their  contributions  to 
the  goal  must  be  listed.  This  will  involve  considerations 
of  constraints,  such  as  time  and  money,  that  may  make  some 
alternatives  infeasible.  Fourth,  one  must  show  how  well 


each  alternative  satisfies  the  goal.  Both  qualitative  and 
quantitative  factors  will  need  to  be  considered.  Last,  one 
alternative  that  best  meets  the  goal  must  be  chosen,  or  if  a 
few  alternatives  may  be  done  they  need  to  be  put  in  a 
priority  order  (16:35-51). 

TAF  completes  most  of  these  five  steps.  They  have 
specified  a  specific  goal  of  maximizing  the  R&M  benefit 
while  also  trying  to  minimize  the  cost.  The  decision 
elements  of  the  hierarchy  tree  are  the  performance  factors 
by  which  the  mods  are  evaluated.  The  alternatives  have  been 
Identified,  but  some  uncertainty  in  both  the  inclusion  of 
all  potential  mods  and  the  data  for  the  mods  does  exist. 

TAF  effectively  shows  how  the  alternatives  satisfy  the  goal 
by  using  a  model  that  assigns  a  value  to  each  mod.  Last, 
the  mods  are  prioritized  by  the  values  assigned  by  the 
model.  The  only  apparent  problem  that  may  exist  is  in  the 
application  of  the  model  itself. 

UaJOna  a.  Decision 

This  section  addresses  the  three  specific  criteria  that 
TAF  needs  to  meet  when  making  their  decision  on  the  mod 
prioritization.  Each  of  these  criteria  will  be  discussed, 
followed  by  an  analysis  of  how  well  TAF  is  meeting  that 
critierla.  The  three  specific  criteria  that  TAF  needs  to 
consider  are  the  acquisition  of  good  data,  identification  of 
all  alternatives,  and  correct  valuation  of  the  potential 
mods.  Each  one  of  these  will  be  discussed  in  detail  and  in 


the  context  of  TAF's  specific  decision.  It  is  vital  that 
TAF  meet  the  requirements  imposed  by  each  of  these  criteria 
to  maximize  the  probability  of  a  good  outcome  of  their 
decision. 

The  data  TAF  uses  must  obviously  be  the  data  necessary 
to  compare  the  mods.  This  means  first  that  data  must  be 
available  for  each  decision  element  in  the  hierarchy  tree 
(l.e.  reliability  improvement  and  mod  cost).  Second,  if  the 
data  is  available,  is  it  '’good*1  data?  In  other  words,  the 
sources  must  be  reliable,  knowledgeable,  motivated,  and  they 
must  understand  exactly  what  TAF  needs  so  they  supply 
accurate  data.  Third,  TAF  must  understand  the  form  of  the 
data.  For  example,  if  they  are  concerned  with  reliability 
Improvement  for  a  certain  situation,  the  data  must  be  a 
measure  of  reliability  improvement  for  the  specific 
situation.  TAF  must  be  able  to  tell  that  the  data  is  indeed 
what  they  were  looking  for. 

TAF  is  able  to  acquire  all  the  data  necessary  to  do  the 
comparisons  among  the  mods.  The  data  is  fairly  good  in  that 
it  comes  from  a  data  system  that  was  designed  to  provide 
information  on  maintenance  and  operational  data  (12:2-1). 
(This  data  set  is  discussed  in  detail  in  Chapter  4  in  the 
section  "Analysis  of  TAF's  Use  of  the  AHP").  As  in  any  data 
set,  there  do  exist  some  deficiencies  and  in  this  case  the 
data  set  has  a  plus  or  minus  10%  error  built-in.  This  is 
due  to  various  factors  such  as  faulty  reporting  and 
paperwork  avoidance  (2).  In  spite  of  this  problem,  the  data 
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set  that  TAF  is  using  is  probably  the  best  source  for  the 
data  they  need.  A  second  problem  with  the  data  acquisition 
is  that  there  exists  a  possibility  that  TAF  does  not 
understand  exactly  what  the  data  means.  Due  to  the  numerous 
definitions  and  measures  of  reliability  and  maintainability, 
for  example,  it  is  difficult  to  really  understand  what  a 
certain  piece  of  data  actually  means.  TAF  simply  needs  to 
work  closely  with  the  people  who  really  understand  the  data 
base  and  extract  that  data  appropriate  for  their  needs. 

The  identification  of  all  the  alternatives,  the 
potential  mods,  is  crucial  for  TAF  to  have  a  valid 
prioritization  of  the  mods.  First,  their  method  of 
collecting  the  mods  must  ensure  that  all  those  who  have 
information  on  mods  are  contacted.  Second,  once  these 
people  are  contacted,  they  must  understand  exactly  what  TAF 
needs  and  why  so  they  understand  the  importance  of  the 
request  for  mods.  Third,  when  all  the  alternatives  have 
been  identified,  they  must  be  defined  specifically  and 
precisely. 

TAF  very  efficiently  identifies  all  the  potential  mods. 
This  is  discussed  in  detail  in  the  previous  chapter  in  the 
section  "TAF's  Process  to  Develop  the  List".  Recall  that 
TAF  contacts  the  using  commands'  system  program  managers  for 
each  of  the  TAF  aircraft.  These  system  program  managers 
receive  the  proposed  mods  for  their  aircraft  from  the  field 
units  and  the  staff  agencies  of  the  major  commands 
(5).  Thus,  both  the  people  who  actually  maintain  the 
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aircraft  and  the  people  who  manage  the  aircraft  are 
contacted  in  the  search  for  mods.  When  the  mods  arrive  to 
those  collecting  them,  they  are  specifically  and  precisely 
defined  by  entries  on  the  data  worksheet.  Each  mod  is 
assigned  a  unique  mod  number  and  an  identification  is  made 
of  the  types  and  number  of  aircraft  on  which  the  mod  may  be 
done.  There  is  also  a  word  description  of  the  mod.  All 
these  descriptions  of  the  mod  provide  the  necessary 
information  needed  to  obtain  correct  data  on  the  mod,  which 
is  ultimately  the  purpose  of  this  requirement  of  precisely 
defining  the  mod. 

The  correct  valuation  of  each  mod  is  the  most  difficult 
criteria  for  TAF  to  meet.  What  this  criteria  means  is  that 
if  mod  A  is  preferred  to  mod  B,  the  process  that  TAF  uses  to 
assign  a  value  to  each  mod  must  give  mod  A  a  value  that  is 
higher  than  the  value  given  to  mod  B.  The  process  that  TAF 
uses  is  the  AHP,  thus  the  AHP  must  yield  values  for  the  mods 
that  are  consistent  with  TAF's  preferences  for  the  mods. 

The  way  to  prove  this  is  by  attempting  to  get  results  from 
the  AHP  that  violate  this  requirement.  Some  possible  areas 
that  may  be  of  concern  are  risk  associated  with  certain 
mods,  uncertainty  in  the  data,  and  dependency  between  the 
mods . 

TAF  has  assigned  a  value  function  to  their  problem 
which  can  be  derived  from  their  hierarchy  tree.  The 


function  is: 


( .301)*<"RIH)  +  ( .301)*<"MIM)  +  < .098)*("IS")  + 

( .201)*("MC")  +  (  .099)*("LC")  =  value 

where  an  entry  in  quotes  means  the  value  assigned  to  mod  x 

In  terms  of  that  decision  element,  and  RI  =  reliability 

improvement,  MI  =  maintainability  Improvement,  IS  = 

inventory  size,  MC  =  mod  cost,  and  LC  =  remaining  life 

cycle.  A  detailed  discussion  of  a  case  where  this  value 

function  does  not  give  a  higher  value  to  the  mod  that  would 

be  preferred  by  TAF  is  presented  in  the  following  chapter  in 

the  section  "Improvement  to  TAF's  Tree".  To  sum  up  the 

results  of  that  example,  mod  A  was  preferred  to  mod  B  since 

mod  A's  life  cycle  cost  savings  were  higher  than  mod  B's, 

but  the  above  value  function  gave  mod  B  the  higher  value. 

In  other  words  mod  A  was  preferred  to  mod  B  but  the  value  of 

mod  A  was  less  than  the  value  of  mod  B.  Thus,  TAF  does  not 

satisfy  this  important  criteria.  However,  the  fix  ,  which 

is  explained  later,  is  an  easy  one. 

To  sum  up  this  section,  TAF  is  doing  a  fairly  good  job 
at  meeting  the  three  decision  criteria  discussed.  There 
exist  some  minor  problems  in  the  data  area  and  a  major 
problem  in  the  valuation  area.  However,  neither  problem  is 
too  difficult  to  remedy  and  should  not  pose  any  large 
problems.  A  little  refinement  is  all  that  is  needed. 

AUE  Applicability 


TAF  uses  the  AHP  to  solve  their  problem  of  prioritizing 
potential  mods.  This  section  will  first  explain  the  type  of 


situation  that  normally  exists  for  a  proper  application  of 
this  process.  The  appropriateness  of  TAF  using  the  AHP  In 
their  specific  situation  will  be  discussed.  Various 
decisions  made  using  the  AHP  will  be  presented  to 
demonstrate  the  process’s  wide  applicability.  This  is 
followed  by  a  brief  discussion  on  the  requirement  that  needs 
to  be  satisfied  before  a  problem  can  be  solved  with  the  AHP. 

The  AHP  is  a  "powerful  process  for  tackling  complex  ... 

problems  (20:22)".  For  this  reason  it  has  been  applied 

in  highly  diverse  areas.  Almost  all  applications  of  the  AHP 

involve  rating  decision  alternatives  and  all  applications 

involve  some  qualitative, as  opposed  to  quantitative, 

elements  that  play  an  essential  role  in  the  decision  problem 

(27:101).  According  to  Grant, 

If  there  are  multiple  alternatives  and  if  the 
irreducibles  need  to  be  given  weight  in  the  final 
choice,  it  is  helpful  to  examine  alternatives  in  pairs. 
It  is  much  easier  to  reach  a  conclusion  about  the  impact 
of  a  variety  of  irreducibles  ...  when  the  decision  maker 
looks  at  only  two  alternatives  at  a  time  18:296) 

"The  pairs  comparison  technique  ,  (the  method  used  in  the 

AHP),  it  is  claimed,  utilizes  the  power  of  human  judgment 

more  effectively  than  ranking  alternatives  directly 

(25)".  Since  the  AHP  is  based  on  the  use  of  paired 

comparisons,  we  see  that  it  deals  well  with  problems  that 

have  qualitative,  or  irreducible,  factors  to  consider.  Most 

applications  are  "complex  in  that  they  involve  a  host  of 


interrelated  elements  with  varying  degrees  of  impact  on  the 
decision  (27:101)". 


TAF's  need  to  prioritize  a  given  set  o£  mods  is  indeed 


a  complex  problem.  However,  it  is  well  suited  for  an  AHP 
application.  Similar  to  most  of  the  other  decisions  that 
were  made  using  the  AHP,  TAF’s  prioritization  involves 
rating  decision  alternatives  (i.e.  the  potential  mods).  It 
involves  both  quantitative  and  qualitative  considerations. 

As  discussed  above,  these  situations  are  ones  that  the  AHP 
is  designed  to  deal  with  effectively. 

The  AHP  has  been  used  to  solve  numerous  diverse 
decision  problems  in  many  different  fields.  For  example,  it 
has  been  used  in  business,  personal  and  domestic  planning, 
public  policy  planning,  and  economic  policy  planning,  to 
name  only  a  few.  In  the  area  of  business,  the  AHP  has  been 
used  to  choose  equipment,  decide  on  leasing  versus  buying, 
choose  a  management  candidate,  decide  on  how  to  stagger 
industry  hours  for  energy  conservation,  and  decide  on  a 
plant  location.  In  the  area  of  personal  and  domestic 
planning,  the  AHP  has  been  applied  to  decisions  on  choosing 
a  car,  a  home  computer,  a  career,  an  ideal  investment,  a 
school,  and  a  house.  (20:37-54)  It  is  evident  that 
there  do  exist  many  different  types  of  decision  problems 
that  can  successfully  be  solved  by  the  AHP. 

After  an  extensive  literature  review,  only  one 
restriction  on  the  application  of  the  AHP  was  found.  An 
article  by  Kamenetsky  states,  "the  AHP  can  only  be  applied 


to  multiple-criteria  decision  problems  in  which  the 
conditions  for  the  existence  of  an  additive  value  function 


are  satisfied  (9:702)**.  The  necessary  condition  for  the 
existence  of  an  additive  value  function  is  that  the 


IV.  Solution  within  the.  AH E 


This  chapter,  assuming  that  the  analytic  hierarchy 
process  Is  a  good  methodology  for  TAF  to  use  in  solving 
their  mod  prioritization,  focuses  on  the  use  of  the  process 
by  TAF.  The  chapter  is  divided  into  three  sections.  The 
first  section  is  a  detailed  description  and  critique  of  the 
separate  steps  of  the  analytic  hierarchy  process  that  TAF 
accomplishes .  Several  problems  that  exist  in  the  TAF 
hierarchy  tree  are  identified.  The  second  section  presents 
a  new  hierarchy  tree  for  TAF  to  use  that  will  correct  some, 
but  not  all,  of  the  problems  with  the  tree  they  are 
currently  using.  The  third  section  presents  a  much  more 
extensive  tree  that  takes  into  consideration  all  of  the  R&M 
2000  goals,  thus  correcting  more  of  the  deficiencies 
associated  with  the  current  TAF  tree. 

Analysis  oi  TAf's  Use  ai  AHE 

The  next  five  subsections  are  an  analysis  of  TAF's  use 
of  AHP .  The  structure  of  their  hierarchy  tree  will  be 
examined  and  potential  problems  with  it  will  be  explained. 
Next,  the  process  of  determining  the  weights  TAF  uses  for 
the  decision  elements  of  the  tree  will  be  discussed  and 
critiqued.  Ideas  on  who  should  determine  the  weights  will 
then  be  presented.  The  collection  of  the  data  and  its 
presentation  to  the  panel  will  be  the  next  topic.  The 
problems  that  exist  in  these  areas  are  highlighted  and 
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possible  solutions  are  offered.  The  fourth  section 
discusses  the  comparison  step  on  the  mods  done  by  the  panel 
members.  Both  positive  and  negative  observations  on  the 
group's  approach  are  presented.  The  last  section  discusses 
the  model  used  by  TAF  to  solve  the  comparison  matrices  in 
order  to  determine  the  value  for  each  mod.  This  section 
also  suggests  ways  the  model  improves  TAF's  final 
prioritization  of  the  mods. 

Tree  Structure .  Recall  that  the  first  step  of  the 
analytic  hierarchy  process  structures  the  decision  by 
constructing  a  tree.  The  objective  of  the  decision  appears 
at  the  top  of  the  tree,  the  decision  elements  occupy  the 
middle  levels,  and  the  alternative  actions  appear  at  the 
lowest  level.  TAF  has  set  up  their  tree  as  depicted  in 
Figure  3. 

Two  problems  exist  with  the  tree  used  by  TAF.  The 

first  problem  Is  caused  by  the  process  TAF  uses  to  develop 

the  tree.  An  article  by  Saaty  states: 

...IThe)  design  of  an  analytic  hierarchy  -  like  the 
structuring  of  a  problem  by  any  other  method  -  is  more 
art  than  science.  It  necessarily  entails  substantial 
knowledge  about  the  system  in  question.  A  very  strong 
aspect  of  the  AHP  is  that  the  individuals  knowledgeable 
about  the  system  . . .  who  supply  judgments  to  make  the 
pairwise  comparisons,  also  play  a  prominent  role  in 
specifying  the  hierarchy  121:205]. 

Thus,  the  panel  members,  who  possess  the  required  knowledge 

about  the  various  systems,  should  be  helping  to  develop  the 

tree.  Unfortunately,  only  one  person  in  TAC  SMO-R&M  does 

this  structuring  of  the  problem.  The  panel  members  only 
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Figure  3.  The  Current  TT'F  Tree 


agree  or  disagree  on  what  Is  presented  to  them.  Although 
the  panel  members  do  have  Input  in  this  way.  It  is  passive 
versus  the  active  participation  advocated  by  Saaty. 

The  second  problem  involves  some  of  the  actual  decision 
elements  in  the  tree.  Specifically,  the  inventory  size  and 
the  remaining  life  of  the  system  are  important 
characteristics  of  the  mods  to  be  considered,  however,  they 
are  directly  tied  in  with  the  "mod  cost"  decision  element. 
For  example,  the  inventory  size  (#  of  aircraft)  will  affect 
the  total  cost  of  the  mod  since: 

Total  mod  cost  =  (mod  cost/aircraft)  *  (#  of  aircraft) 
Thus,  the  greater  the  Inventory,  the  more  money  is  spent. 
However,  TAF  currently  considers  a  large  inventory  size  as 
desirable  since  the  research  and  development  cost  per  mod 
will  be  reduced.  Looking  at  the  total  mod  cost,  TAF  should 
prefer  a  small  inventory  size,  but  looking  at  only  the 
research  and  development  cost,  TAF  should  prefer  a  large 
inventory  size.  Which  one  is  the  right  preference?  This 
confusion  is  easily  eliminated  by  simply  realizing  what 
TAF’s  ultimate  goal  is  when  considering  the  cost  of  a  mod. 
They  want  to  minimize  the  cost.  Thus,  TAF  should  consider 
the  total  mod  cost  and  prefer  the  mod  with  the  smaller  total 
mod  cost. 

Next,  consider  the  decision  element  of  "remaining  life 
of  the  system".  This  element  also  directly  relates  to  cost 
in  the  following  manner: 


Total  LCC  savings  =  (LCC  savings  per  year)  *  (remaining 
life)  where  LCC  is  life  cycle  cost  savings.  If  TAF  is 
concerned  with  the  cost,  or  In  this  case  the  cost  avoidance, 
then  the  total  LCC  savings  Is  a  better  measure  of  the  amount 
saved  than  the  remaining  life  of  the  system.  For  example, 
consider  two  mods.  Mod  A  may  save  $10/year  and  its 
remaining  life  Is  10  years.  Thus,  its  total  LCC  savings  is 
$100.  Mod  B  may  save  $20/year,  but  its  remaining  life  is 
only  5  years.  Mod  B's  total  LCC  savings  Is  $100.  These 
mods  would  be  rated  equal  in  terms  of  total  LCC  savings, 
which  TAF  Is  ultimately  concerned  with,  but  mod  A  would  be 
preferred  to  mod  B  if  only  the  remaining  life  was 
considered.  It  is  Important  to  take  into  account  both  the 
amount  saved  per  year  as  well  as  the  remaining  life  of  the 
system,  and  both  of  these  are  embodied  in  the  total  LCC 
savings  calculation. 

Having  accepted  the  new  ideas  of  total  mod  cost  and  LCC 
savings  as  being  the  data  that  TAF  should  actually  be 
considering,  there  is  one  more  step  to  be  done.  The  present 
value  of  both  the  life  cycle  cost  savings  and  the  mod  cost 
must  be  calculated.  Then  the  present  value  mod  cost  will  be 
subtracted  from  the  present  value  total  LCC  savings.  This 
figure  will  yield  the  net  present  value  life  cycle  cost 
savings.  This  figure  would  be  the  only  one  the  panel  would 
need  to  consider  to  accurately  take  the  mod  cost,  inventory 
size,  and  remaining  years  left  for  the  system,  into  account 


(The  details  of  using  this  approach  will  be  addressed  in  the 
section  "Improvement  to  TAF's  Tree"). 

Tree  Weights .  Once  the  decision  elements  have  been 
decided  upon  and  the  decision  hierarchy  tree  has  been  set 
up,  the  next  important  step  is  to  decide  the  weights  for  the 
branches  on  the  tree  which  are  associated  with  the  decision 
elements.  Currently,  prior  to  the  panel  meeting  where  the 
comparisons  between  the  mods  are  made,  people  from  the  R&M 
shops  for  the  various  Tactical  Air  Force  aircraft  coordinate 
on  the  weights  for  the  decision  elements.  These  weights  are 
decided  upon  by  a  single  person  in  TAC  SMO-R&M  (5).  This 
person  is  not  the  actual  decision  maker,  but  rather  the 
analyst  who  reports  to  the  decision  maker.  However,  the 
choice  of  the  weights  for  the  decision  elements  is  a  task 
that  should  be  accomplished  by  the  decision  maker  who  has 
the  ultimate  responsibility  for  the  prioritized  list  that  is 
developed.  By  developing  the  weights,  the  decision  maker 
has  fixed  the  relative  importance  he  feels  exists  between 
the  various  decision  elements.  In  other  words,  he  has  Input 
information  to  the  AHP  that  Indicates  his  opinion  on  the 
Importance  of  the  various  decision  elements.  Thus,  although 
he  will  not  be  involved  with  the  time  consuming  task  of 
comparing  the  data  on  the  mods,  (this  is  done  by  the  panel 
memebers),  he  has  not  lost  his  decision  making  authority 
because  he  has  already  had  his  turn  at  structuring  the 
problem.  Due  to  this  fact,  the  decision  maker  will  be  more 
inclined  to  accept  the  results  of  the  analysis. 


Looking  ahead  to  the  section  "Improvement  to  TAF's 
Tree",  and  assuming  that  TAF  accepts  this  new  hierarchy 
structure,  one  sees  that  there  are  only  two  decisions  on 
weights  that  the  decision  maker  needs  to  make.  First,  he 
must  decide  how  much  importance  to  attach  to  R&M  improvement 
versus  the  overall  cost  of  the  mod.  Second,  he  needs  to 
decide  how  much  weight  to  attach  to  reliability  improvement 
versus  maintainability  improvement.  Even  though  there  are 
only  these  two  sets  of  weights  to  determine,  it  is  very 
possible  that  the  decision  maker  may  not  be  completely  sure 
of  the  weights  he  assigns.  This  should  not  pose  a  big 
problem  since  sensitivity  analysis  can  be  done.  In  a  later 
section,  "Analysis  of  TAF's  U3e  of  the  AHP " ,  the  sensitivity 
analysis  that  is  performed  on  the  Planning  and  Programming 
Priority  Project  (P4)  model  is  discussed.  Thus,  the 
decision  maker  can  specify  a  weight  of  .7  plus  or  minus  .1 
for  the  R&M  Improvement  and  .3  plus  or  minus  .1  for  the 
overall  cost.  The  P4  model  would  then  be  run  using  the 
following  weights: 

Run  1  Run  2  Run  3 

R&M  improvement  .7  .6  .8 

Overall  cost  .3  .4  .2 

The  results  for  each  run  (l.e.  the  prioritized  list  of  mods) 
would  then  be  available  for  the  decision  maker  to  make  his 
final  determination  of  the  proper  prioritization  of  the 


Data  and.  Its  Presentation  £&  the.  Panel.  Prior  to  the 
meeting  for  the  panel  members,  TAF  completes  worksheets 
containing  information  on  R&M  Improvement,  R&M  ranking, 
inventory  size,  mod  cost,  and  the  remaining  life  of  the 
system.  The  first  two  areas  of  information  are  requested 
from  the  Maintenance  and  Operational  Data  Access  System 
(MODAS).  Inventory  size  and  mod  cost  are  figures  supplied 
from  the  Air  Logistics  Centers  functional  managers.  Data  on 
the  remaining  life  of  the  system  is  contained  in  the  TAC/XP 
planning  documents  (5).  This  information  is  then  used  in 
the  comparison  of  the  mods.  This  sounds  very 
straightforward,  but  there  are  some  critical  problems  that 
exist  in  both  the  data  and  the  manner  in  which  it  is 
presented  to  the  panel  members. 

The  primary  problems  in  the  data  collection  lie  in 
MODAS  itself  and  the  interpretation  of  its  data  by  TAF. 

MODAS  is  a  "central,  on-line,  interactive  data  storage  and 
access  system  for  the  storage  and  processing  of  maintenance 
and  operational  data  on  selected  USAF  weapon  systems 
(12:2-1)".  MODAS  unfortunately  has  a  plus  or  minus  10% 
built-in  error  (2).  Thus,  the  panel  members  may  be  misled 
if  they  fail  to  recognize  this  fact.  (A  discussion  of  this 
appears  in  Chapter  5). 

The  area  of  R&M  Improvement  is  considered  so  the  panel 
member  can  determine  which  mods  will  supply  the  most 
Increase  in  R&M.  For  example,  when  comparing  two  mods  on 
reliability  Improvement,  the  panel  considers  the  difference 


between  the  reliability  o£  the  component  prior  to  the  mod 
versus  the  projected  reliability  of  the  component  after  the 
mod.  The  mod  with  the  biggest  improvement  is  thus  the 
preferred  mod.  TAF  uses  the  mean  time  between  maintenance 
(MTBM)  in  this  comparison.  They  must  realize  that  MTBM 
measures  only  the  logistics  reliability  whereas  mean  time 
between  critical  failure  (MTBCF)  measures  the  operational 
reliability  (2).  Depending  on  what  TAP'  feels  is  more 
important,  they  may  want  to  reconsider  which  measure  of 
reliability  improvement  to  use.  When  considering 
maintainability  improvement,  TAF  looks  at  the  maintenance 
manhours  per  flying  hour  ( MMH/FH ) .  Unfortunately,  MMH/FH 
data  is  not  valid  at  the  component  level  (2).  The  measure 
TAF  should  use  is  mean  time  to  repair  ( MTTR )  for  the 
component,  but  MODAS  only  supplies  this  data  indirectly. 

The  actual  figure  must  be  calculated  by  hand  after  drawing 
out  the  appropriate  data  from  MODAS  (2).  Thus,  TAF  may  not 
be  pulling  out  the  appropriate  data  from  MODAS  to  do  their 
mod  comparisons  on  R&M  improvement. 

The  area  of  R&M  ranking  is  included  to  provide  the 
panel  member  with  Information  on  how  critical  the  component 
is  to  the  R&M  of  the  system.  Thus,  it  component  A  is  more 
critical  to  its  system  than  component  B,  (i.e.  component  A's 
improvement  would  increase  the  reliability  or 
maintainability  of  its  system  more  than  the  improvement  of 
component  B)  then  mod  A  will  carry  more  weight  and  be 
preferred  over  mod  B.  MODAS  provides  information  for  the 


worst  systems,  worst  50  subsystems,  and  worst  100  work  unit 
codes,  which  are  referred  to  in  this  discussion  as 
components.  The  data  is  generated  from  the  latest  3-month 
fleet  MTBM  for  the  reliability  ranking  and  by  MMH/FH  for  the 
maintainability  ranking  (12:8-2).  A  component  having  a 
ranking  of  1  would  be  the  component  in  the  system  that 
causes  the  system  to  fail  the  most.  This  is  done  looking  at 
both  the  reliability  of  the  system  and  also  at  its 
maintainability.  TAF's  idea  to  use  the  MODAS  data  is  good, 
except  that  it  overlooks  one  important  point.  Namely,  if  a 
component  is  ranked  as  the  worst  component  in  terms  of 
reliability  and  mod  A  is  done  on  it  that  projects  to 
Increase  the  reliability  by  400%,  perhaps  it  would  still  be 
the  worst  component  in  terms  of  reliability.  However,  a 
different  component  may  be  the  second  worse  in  terms  of 
reliability  and  by  performing  mod  B,  thereby  Increasing 
MTBM  by  only  10%,  it  falls  into  the  ninth  worse  ranking 
position.  In  this  situation  TAF  would  have  given  mod  A  more 
weight  than  mod  B  since  mod  A  had  a  worse  ranking.  However, 
mod  B  would  probably  be  the  better  choice  and  should  have 
been  given  more  weight  than  mod  A.  Thus,  it  is  important  to 
consider  how  critical  the  component  is,  as  well  as  to 
consider  how  much  its  criticality  will  be  reduced  by 
performing  the  mod. 

Fortunately,  MODAS  provides  a  table  that  makes  this 
task  straightforward.  The  data  in  the  following  table  can 


easily  be  taken  from  the  Worst  Case  Reliability  Report 
(12 : A3-2 ) . 


Component 

MTBM ( * ) 

Ranking 

1 

100 

1 

2 

200 

2 

3 

300 

3 

(*)  old  MTBM  measured  in  hours 

In  this  case,  component  1,  with  an  MTBM  of  100  hours,  is 
ranked  as  the  worst  component  in  the  system  because  it 
causes  the  system  to  experience  a  "type  one"  failure  most 
often  (i.e.  every  100  hours)  (12:8-2).  One  need  only 
consider  the  new  projected  MTBM  for  a  component  and  see  if 
the  ranking  for  that  component  will  improve.  Bor  example, 
if  you  are  considering  component  1  and  it  has  a  projected 
new  MTBM  of  200  hours,  then  its  ranking  has  moved  to 
position  2.  This  is  an  improvement  of  1.  However,  if  the 
new  MTBM  is  150,  it  will  not  change  its  ranking  at  all. 

Thus,  the  panel  should  not  only  consider  the  current 
ranking,  but  also  the  new  ranking  for  the  component  after 
the  proposed  mod  is  accomplished. 

A  secondary  problem  of  the  data  collection  rests  in  the 
information  gathered  for  the  mod  costs.  The  per  year  costs 
will  be  assumed  to  be  good,  but  they  are  not  put  into 
present  value  terms.  This  may  cause  faulty  cost 
comparisons  to  be  made  by  the  panel.  (Details  of  the 


problem  this  creates  will  be  discussed  in  the  section 
"Improvement  to  TAF's  Tree"). 

When  the  data  is  collected  it  is  provided  to  all  the 
panel  members  in  the  form  depicted  in  Figure  4.  This  form 
makes  it  clumsy  for  the  panel  members  to  compare  the  mods. 

A  preferred  format,  containing  the  same  information  but 
making  it  easier  to  compare  the  mods  is  shown  in  Figure  b. 
Notice  that  all  the  potential  mods  are  listed  down  the  left 
side  of  the  sheet.  The  mod  name  and  mod  number  are  listed 
across  the  top.  The  entries  of  life  remaining,  number  in 
Inventory,  reliability  Improvement  and  reliability  ranking, 
maintainability  Improvement  and  maintainability  ranking,  and 
total  cost  would  separately  be  entered  in  the  row  entry 
titled  "decision  element".  Thus,  there  would  be  one  data 
worksheet  for  each  decision  element.  This  format  has  the 
advantage  of  providing  all  the  Information  needed  for  each 
comparison  on  a  single  sheet  of  paper.  Thus,  the  members 
can  quickly  discern  the  high  and  the  low  values  and  have  a 
more  solid  grasp  of  the  range  of  the  values  involved.  This 
is  critical  for  the  consistency  of  the  comparisons  done  with 
the  analytic  hierarchy  process.  (This  concept  will  be 
discussed  in  detail  in  Chapter  5). 

Comparison  Step .  Armed  with  the  data  on  the  mods  and  a 
hierarchy  tree  with  all  the  decision  elements  weighted,  the 
panel  is  ready  to  make  the  pairwise  comparisons  of  the  mods. 
There  are  several  points  to  be  made  about  the  panel  and  how 
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Figure  5.  New  Worksheet  for  the  Data 


It  functions.  First,  the  panel  consists  of  people  who  are 
familiar  with  the  various  TAF  aircraft  systems  on  which  the 
mods  will  be  done.  They  are  the  "experts".  The  panel 
assembles  in  a  conference  room  where  they  go  through  all  the 
necessary  comparisons  between  the  mods.  Each  panel  member 
is  given  the  chance  to  say  what  his  score  is  for  each 
comparison,  for  example  that  mod  A  is  moderately  better  than 
mod  B  in  terms  of  reliability  improvement.  He  is  also  given 
the  chance  to  explain  his  reasons  for  the  score.  When  every 
member  has  had  his  turn,  the  panel  chairman  assigns  the 
final  score  for  that  comparison  (5). 

The  manner  in  which  the  panel  operates  has  some 

positive  points  and  some  negative  ones.  Looking  first  at 

the  positive  side,  an  article  by  Quade  states  that  : 

When  working  with  individual  experts,  it  is  important  to 
insist  that  each  one  make  the  logic  behind  his  opinions 
or  judgments  explicit.  For  only  when  the  reasoning  is 
explicit  can  someone  else,  whose  information  and 
perspective  may  be  different,  use  the  work  of  the  first 
to  modify  his  own  opinion  (16:328). 


Since  each  member  of  the  panel  can  not  possibly  be  aware  of 
all  the  details  of  all  the  TAF  aircraft  systems,  the  open 
exchange  of  rationale  for  scoring  mods  performed  by  the 
panel  is  Indeed  necessary.  Thus,  TAF  is  right  on  the  mark 
in  this  respect. 

Another  comment  by  Quade  on  the  idea  of  using  experts 
efficiently  states  that  providing  an  effective  means  of 
communication  among  experts  is  critical.  A  common  frame  of 
reference  is  important  to  this  communication  and  an  ideal 
way  to  enforce  it  is  by  presenting  the  problem  to  the 
experts  "in  terms  of  an  analytic  model,  or  even  better,  to 
have  them  participate  in  formulating  the  model  as  well 
(16:329)".  The  model  in  TAF ' s  case  is  the  hierarchy  tree 
within  the  AHP .  Although  currently  the  panel  does  not  have 
a  lot  of  input  in  developing  the  model,  they  do  understand 
its  meaning  (5).  The  fact  that  a  model  exists  at  all  is  the 
important  point  to  consider,  since  it  enables  TAF  to  fulfill 
the  need  for  a  common  frame  of  reference.  Each  decision 
element  in  the  tree  is  explicitly  defined  and  explained  to 
each  panel  member  so  that  in  a  discussion  about  reliability 
improvement,  for  example,  everyone  is  talking  about  the  same 
thing.  However,  on  the  negative  side,  Quade  goes  on  to  say 
that : 

Experiments  have  shown  that  the  best  use  of  a  number  of 
experts  is  not  the  traditional  method  of  having  the 
Issues  presented  to  them  and  debated  in  open  round-table 
discussion  until  a  consensus  emerges  or  until  they 
arrive  at  an  agreed  upon  group  position  116:328). 


He  feels  that  committees  often  fall  to  make  their 


assumptions  and  reasoning  explicit  and  that  one  needs  to 
avoid  the  psychological  drawbacks  of  a  round-table 
discussion  (16:328).  Since  the  TAF  panel  performs  in  the 
round-table  setting  described  by  Quade,  there  is  the  chance 
for  some  undesirable  outcomes.  For  example,  some  members' 
opinions  may  be  swayed  by  others  who  are  more  verbal  about 
their  opinions,  but  who  also  unfortunately  may  not  have  all 
the  pertinent  facts.  The  panel  can  not  be  performing 
optimally  if  this  happens. 

Getting  Logical  Answers .  Recall  that  the  AHP  will 
yield  a  value  under  each  decision  element  for  each  mod  using 
the  data  generated  from  the  pairwise  comparisons.  These 
values  are  then  aggregated  up  the  hierarchy  producing  an 
overall  value  for  each  mod.  These  final  values  are  then 
used  to  rank  the  mods  (22).  TAF  uses  the  Planning  and 
Programming  Priority  Project  (P4)  model  to  solve  for  the 
values  of  each  mod. 

The  P4  model  supplies  two  types  of  information.  First, 
the  mods  with  their  corresponding  values  are  given.  Second, 
the  consistency  ratio  at  each  decision  element  node  and  the 
overall  consistency  ratio  indicates  how  good  the  comparisons 
are.  A  value  of  less  than  .1  is  a  good  consistency  ratio 
(22).  To  illustrate  consistency,  consider  a  comparison  of 
mod  A,  mod  B,  and  mod  C.  If  the  panel  states  that  mod  A  is 
equal  to  mod  B,  and  that  mod  A  is  moderately  better  than  mod 


C,  then  they  should  say  that  mod  B  is  moderately  better  than 
mod  C.  If  they  do  this  their  consistency  ratio  will  be  0 
(the  best  value),  but  if  they  say  mod  B  is  equal  to  mod  C 
their  ratio  will  get  worse.  Thus,  the  panel  can  make  all 
their  comparisons  and  P4  will  highlight  the  comparisons  (if 
any)  that  are  unsatisfactory  by  yielding  a  high  consistency 
ratio  (i.e.  >.l).  The  panel  can  then  go  back  to  their 
comparison  and  check  their  work. 

The  P4  model  also  has  the  capability  to  easily  change 
the  structure  and/or  the  weights  of  the  tree.  This  allows 
for  sensitivity  analysis  to  be  done  quickly  (22).  Thus,  if 
there  exists  some  doubt  as  to  what  the  weight  of  inventory 
size  should  be,  the  P4  model  can  show  graphically  what  new 
weights  (higher  or  lower)  for  inventory  size  would  change 
the  ranking  of  the  mods.  Consider  that  the  inventory  size 
weight  is  currently  set  at  .2  and  you  have  a  plus  or  minus 
.05  bound  on  this.  If  you  must  change  it  to  .7  or  .01  to 
change  the  mod  ranking,  then  the  panel  would  not  need  to  be 
too  concerned  if  they  put  the  weight  at  .15,  .2,  or  .25  as 
the  weight  for  inventory  size. 

Due  to  the  facts  that  TAF  uses  a  model  to  calculate  the 
values  for  the  mods,  that  a  consistency  check  is  done,  and 
that  sensitivity  analysis  is  performed,  the  final  values  for 
the  mods,  and  thus  their  ranking,  will  be  satisfactory. 

This  assumes  that  the  P4  model  does  in  fact  solve  the 
comparison  matrix  data  correctly,  and  that  the  data  used  in 
the  comparisons  is  good. 


Improvement  ta.  TAF's  Xrs.e 

This  section  will  present  a  new  hierarchy  tree  for  TAJ' 
to  use  in  prioritizing  its  mods.  The  structure  of  the  tree, 
as  well  as  the  rationale  for  the  structure,  will  be 
discussed.  The  additional  data  required  by  this  tree  will 
be  explained  and  possible  data  sources  mentioned.  Last,  an 
example  of  the  difference  in  the  values  of  the  mods  using 
TAF's  current  tree  and  the  new  tree  will  be  illustrated. 

Structure .  The  structure  of  this  new  tree  (NEWTREE), 
Figure  6,  is  very  similar  to  the  TAF  tree  ( TAFTREE ) .  The 
main  goal  of  the  decision  remains  the  same:  to  receive  the 
highest  R&M  improvement  with  the  least  cost.  The  next  level 
is  also  the  same.  This  level  consists  of  the  two  decision 
elements  "overall  benefit"  and  "overall  cost".  However, 
"overall  benefit"  has  been  renamed  "R&M  improvement".  This 
is  where  the  similarities  end. 

The  decision  elements  "inventory  size",  "remaining  life 
cycle",  and  "mod  cost"  have  been  eliminated  from  TAFTREE. 

The  information  they  contain  has  not  been  lost,  but  rather 
included  in  the  "overall  cost"  decision  element.  The 
"overall  cost”  reflects  the  net  life  cycle  cost  savings 
realized  by  performing  the  mod.  The  details  are  explained 
later  In  the  data  section.  The  decision  elements  of 
"reliability  improvement"  and  "maintainability  improvement" 


remain,  however,  their  interpretation  is  different.  TAFTREE 
considers  the  predicted  improvement  in  R&M  that  is  supplied 
by  the  people  proposing  the  mods.  Therefore,  if  mod  A 


Improved  Its  MTBM  by  100  hours  and  mod  B  improved  its  MTBM 
by  50  hours,  then  mod  A  is  twice  as  good  as  mod  B  in  terms 
of  reliability  improvement.  NEWTREE  will  consider  the 
predicted  improvement  of  the  component  while  also 
considering  the  required  improvement  (i.e.  the  improvement 
in  the  MTBM  of  the  component  which  will  cause  it  to  not  be  a 
critical  player  in  the  MTBM  of  the  system).  Thus,  if  mod  A 
needed  50  hours  improvement  in  MTBM  and  it  achieved  100 
hours,  there  is  an  overkill  of  50  hours.  In  addition,  if 
mod  B  needed  50  hours  improvement  in  MTBM  and  it  achieved 
this  figure,  then  there  would  be  no  overkill.  NEWTREE  would 
rate  mod  A  equal  to  mod  B  in  this  scenario  even  though  mod  A 
has  a  greater  increase  in  MTBM.  The  determination  of  the 
required  MTBM  improvement  will  be  discussed  later  in  the 
data  section. 

Data .  To  use  the  NEWTREE  versus  the  TAFTREE  structure, 
TAF  will  need  a  few  more  pieces  of  data  and  a  new  way  of 
looking  at  data  they  already  have.  The  later  case  occurs 
for  the  "reliability  improvement"  and  "maintainability 
improvement"  decision  elements.  Take  "reliability 
improvement"  for  example.  Currently  TAF  considers  the  hours 
improvement  in  MTBM  and  the  reliability  ranking  of  the 
component  in  the  system.  Consider  components  A,  B,  and  C 
which  are  all  in  the  same  system.  The  following  table 
supplies  the  data  known  about  each  component.  The  MTBM 
hours  are  the  absolute  MTBM  values  for  each  component. 


Assume  that  component  A  Is  the  only  one  with  a  potential 
mod . 

Current 


Reliability 

Ranking 

MTBM 

(current  hours) 

MTBM 

(predicted  hours) 

Component 

A 

1  * 

10 

100 

Component 

B 

2 

60 

60 

Component 

C 

3 

75 

75 

*  »  worst  case 

TAFTREE  looks  at  the  reliability  ranking  and  at 
MTBM( predicted ) -MTBM ( cur rent ) .  This  is  the  only  Information 
they  use  to  make  the  comparisons  among  the  mods.  NEWTREE 
will  look  at  the  reliability  ranking  and  see  what 
improvement  in  MTBM  will  improve  Its  ranking  (i.e.  give  it  a 
higher  ranking).  For  example,  an  Improvement  of  51  hours 
for  component  A  will  move  It  Into  position  2  In  the 
reliability  ranking.  An  improvement  of  66  hours  will  move 
component  A  Into  the  third  position.  Someone  will  need  to 
determine  what  change  In  the  MTBM  of  the  system  Is  desired 
as  the  new  required  standard.  Then,  a  specific  mod  for  that 
system,  with  its  predicted  improvement  in  its  MTBM,  will  be 
evaluated  on  how  well  it  achieves  this  new  standard  (i.e. 
the  system  MTBM).  Say  the  decision  maker  decides  that  a 
system  MTBM  of  60  hours  is  the  new  required  standard.  In 
this  case  component  A  would  have  90  hours  improvement  in 
reliability,  but  only  51  hours  are  required.  Thus,  there  is 


an  overkill  of  39  hours.  NEWTREE  will  not  consider  the 
overkill,  but  rather  take  the  51  hours  as  the  reliability 
improvement  for  component  A.  A  graph  may  explain  this  point 
better.  Consider  the  graph  below  where  the  possible  MTBM 
hours  for  component  A  are  shown  on  the  horizontal  axis  and 
the  required  MTBM  Improvement  (i.e.  the  amount  needed  to 
move  past  the  next  critical  component)  is  shown  on  the 
vertical  axis.  The  data  point  for  the  reliability 
improvement  of  component  A  that  TAF  will  use  to  make  the 
pairwise  comparisons  is  the  maximum  possible  point  on  the 
possible  MTBM  line  (in  this  case  point  "A").  Note  that  this 
line  does  not  extend  all  the  way  to  90  hours  MTBM,  but  is 
cut  short  by  the  required  amount  of  51  hours  MTBM. 


90 

Required 

MTBM 

for 

System 

51 


MTBM  hours 


Note  that  although  this  analysis  only  gives  credit  for 
partial  mods  (i.e.  it  disregards  the  improvement  that  is 
overkill),  the  mod  would  still  have  the  full  predicted 
Improvement  in  MTBM.  The  data  for  the  reliability  ranking 


and  MTBM(current )  Is  available  In  modas  and  the 


MTBM( predicted )  is  submitted  by  the  system  program  managers 
The  same  is  true  for  the  maintainability  improvement  data. 

Another  set  of  data  needed  for  NEWTREE  arises  in  the 
Noverall  cost"  decision  element.  It  is  here  that  some  new 
data  will  need  to  be  obtained.  This  value  for  each  mod  is 
defined  as:  Net  present  value  of  life  cycle  cost  savings= 

PV  (Total  life  cycle  cost  savings)  -  PV  (Total  mod  cost) 
where  PV  stands  for  present  value.  The  mod  cost  per 
aircraft  per  year  figures  are  already  available,  so  it  is  a 
simple  matter  to  calculate  its  present  value.  The  PV  total 
mod  cost  * 


n  r 

Z[' 


cost/AC/ 


fear  ( 1 )  * 
(1.07)0 


aircraft ! 


mod  cost/AC/year ( 2 )  * 
(1.07)* 


alrcrai 


. . .  + 


mod  cost/AC/year ( n )  *  (I  aircraft 
(1.07)  ri-i 


where  n  *  •  of  years  to  do  the  mod  and  AC  =  aircraft.  The 
discount  rate  used  in  this  calculation  is  7%.  The  rationale 
for  using  this  value  is  discussed  below.  Note  that  if  the 
constant  total  mod  cost  was  calculated,  the  equation  would 
be : 

constant  total  mod  cost® 
r\ 

FI  j(  mod  cost/a  ircraft/year  ( 1 ) )  Ml  aircraft)! 

irl  L  -* 

where  n=  I  of  years  to  do  the  mod. 


The  life  cycle  cost  savings  are  a  bit  trickier. 

However,  the  item  managers  could  calculate  the  dollars  saved 

per  year  as  a  result  of  the  mod  (5),  although  they  have  not 

done  this  in  the  past.  Taking  this  figure  and  multiplying 

it  by  the  inventory  size  (i.e.  tt  aircraft)  will  yield  life 

cycle  cost  savings  per  year  for  all  the  aircraft.  Knowing 

the  remaining  life  cycle  for  the  various  aircraft  types,  one 

could  then  discount  the  stream  of  yearly  savings  back  to  the 

present  value  amount.  This  figure  would  be  the  present 

value  of  the  life  cycle  cost  savings. 

present  value  total  life  cycle  cost  savings= 

(LCC  savings/year/mod )*( tt  aircraft)*(P |A,7%,n) 

where  n=  tt  years  left  for  the  aircraft 
( P | A, 7%, 10 )  =  7.024 
(P | A, 7%, 20 )  «  10.594 
(P  |A, 7%, 30 )  «  12.409 

However,  the  constant  total  life  cycle  cost  savings= 

(LCC  savings/year/mod )*( tt  aircraft )*(tt  years) 

The  final  step  calculates  the  net  PV  LCC  savings  which 

is  calculated  by  (PV  total  LCC  savings  -  PV  total  mod  cost). 

The  net  constant  LCC  savings  is  calculated  by  (Constant 

total  LCC  savings  -  Constant  total  mod  cost). 

The  only  other  new  piece  of  data  needed  is  the 

appropriate  discount  rate  to  use  when  calculating  the  net 

present  value  of  the  life  cycle  cost  savings  and  the  mod 

cost.  According  to  Lt  Col  Russel, 

The  Treasury  borrowing  rate  on  long-term  securities  is 
correctly  the  time  value  of  money  for  defense  analysis. 
With  this  position,  today's  ...  correct  rate  for 
discounting  constant  dollar  streams  is  7.0%  (18:9). 


50 


improvement .  Using  the  NEWTREE  versus  the  TAFTREE 
structure  has  three  advantages.  First,  since  there  are 


fewer  decision  elements  to  be  considered  for  each  mod,  there 
are  fewer  comparisons  that  need  to  be  made  by  the  panel 
members.  Second,  the  "political  factors"  of  "inventory 
size"  and  "remaining  life  cycle"  are  incorporated  into  the 
decision  element  of  "overall  cost'  and  thus  are  not  as 
visible.  The  third,  and  the  most  important,  advantage  is 
that  the  prioritized  list  of  mods  more  accurately  reflects 
TAF's  goal  of  minimizing  cost  by  correctly  dealing  with  the 
mod  cost  and  the  life  cycle  cost  savings.  In  other  words, 
NEWTREE  considers  the  present  value  of  the  net  LCC  savings 
versus  TAFTREE  which  only  considers  the  constant  LCC 
savings.  Following  is  a  sample  comparison  of  the 
prioritized  lists  generated  from  the  TAFTREE  and  the  NEWTREE 
structures.  Note  that  the  weights  used  for  the  decision 
elements  are  the  same  in  the  two  trees. 

The  data  shown  below  which  was  used  tor  the  tour  mods 
presented  in  this  example  is  representative  of  real  world 
data.  The  equations  for  calculating  the  constant  dollar  and 
present  value  amounts  presented  earlier  were  used  to  arrive 
at  some  of  the  values  in  the  following  table.  Using  this 
data,  the  following  comparison  matrices  for  the  TAFTREE  and 
NEWTREE  structures  were  developed.  These  matrices  are 
developed  by  using  the  teeter-totter  shown  below. 


Table  1.  Data  Used  to  Show  Improvement 


Hod  1 

Mod  2 

Mod  3 

Mod  4 

1  of  planes 

98 

654 

367 

200 

life  remaining 

20 

20 

10 

30 

(years ) 

reliability  improvement 

( hours ) 

raw 

2 

50 

600 

400 

need 

2 

50 

100 

100 

reliability  ranking 

4 

10 

25 

100 

maintainability  Improvement 

(hours ) 

raw 

10 

- 

200 

50 

need 

10 

- 

100 

50 

maintainability  ranking  30 

40 

2 

25 

mod  cost/aircraf t/year ( i ) 

(constant /present 

value ) 

1 

2/2 

4/4 

10/10 

10/10 

2 

2/1.86 

2/1.86 

2/1.86 

10/9.35 

3 

5/4.37 

1/.87 

2/1.75 

5/4.37 

4 

10/8.16 

1/.8  2 

2/1.63 

2/1.63 

5 

10/7.63 

-/- 

1/.76 

2/1.53 

total  mod  cost 

( constant /present 

value ) 

29M/24.02M 

8M/7.55M 

17M/16M 

29M/26.87M 

life  cycle  cost 

savings /year /mod 

2M 

10M 

8M 

5M 

total  life  cycle 

cost  savings 

constant 

3920M 

130800M 

29360M 

30000M 

present  value 

2076 . 4M 

69284 .6M 

20622M 

12409M 

net  cost  savings 

constant 

1078M 

125568M 

23121M 

24200M 

present  value 

-276M 

64346M 

1 4  7  SOM 

7035M 

NEWTREE  Comparison  Matrices 


RI  1  2  3  4 


MI  1  2  3  4 


1 

2 

3 

4 


E  1/S  1/A  1/A 

S  E  1/S  1/S 

A  S  E  E 

A  S  E  E 


E  1/A  1/S 

E  1/A  1/S 

AES 
S  1/S  E 


COST  12  3  4 


1 

2 

3 

4 


E  1/A  1/M  1/E-M 

A  E  VS  A 

M  1/VS  E  E-M 

E-M  1/A  1/E-M  E 


Note  that  RI  stands  for  reliability  improvement,  MI  for 
maintainability  improvement,  IS  for  inventory  size,  MC  for 
mod  cost,  LC  for  life  cycle  remaining,  and  COST  for  total 
net  life  cycle  cost  savings. 

Using  a  software  package  called  Expert  Choice  (7), 
these  matrices  were  solved  to  yield  the  values  for  the  mods. 
A  table  depicting  the  mods  and  their  associated  values  at 
several  of  the  decision  elements  is  provided  below.  For 
each  decision  element,  the  values  for  all  of  the  mods  will 
sum  to  one.  The  highest  value  for  each  decision  element  is 
given  to  the  mod  that  is  best  in  terms  of  that  decision 
element . 

Notice  that  the  final  prioritization  of  the  mods  is  the 
same  for  both  trees,  but  the  values  for  the  four  mods  are 
different.  Thus,  although  in  this  case  the  mod  ranking  is 
the  same,  it  is  possible  that  for  another  example  the 
ranking  (based  on  the  mod  values)  could  be  reordered.  The 
reason  for  the  different  mod  values  is  a  consequence  of  both 

I 


a  different  tree  structure  and  the  reinterpretation  of  some 
of  the  decision  elements  within  the  tree.  One  sees  the  most 
change  in  the  values  of  the  mods  under  the  "overall  cost” 
decision  element  node  since  this  is  where  the  most  drastic 
adjustments  were  made  to  the  TAFTREE  structure. 


Decision  TAFTREE  NEWTREE 

Element 


mod 

value 

mod 

value 

RI 

3 

.640 

3 

.424 

4 

.235 

4 

.424 

1 

.063 

2 

.114 

2 

.063 

1 

.037 

MI 

3 

.724 

3 

.669 

4 

.166 

4 

.220 

1 

.055 

1 

.055 

2 

.055 

2 

.055 

COST 

2 

.471 

2 

.721 

4 

.234 

3 

.141 

3 

.177 

4 

.084 

1 

.118 

1 

.055 

final  goal 

3 

.481 

3 

.425 

2 

.244 

2 

.276 

4 

.200 

4 

.251 

1 

.075 

1 

.049 

EUI-UieL  ImBE-Q.Y£ffleiUL  tfi  MBMTB.eE. 

A  further  enhancement  to  NEWTREE  would  incorporate  the 
idea  of  aircraft  mission  importance  into  the  structure  of 
the  decision.  This  will  allow  for  two  changes.  First,  the 
mission  Importance  of  each  aircraft  relative  to  the  others 
may  be  shown.  Second,  the  "reliability  improvement"  and 


"maintainability  improvement"  decision  element  weights  may 
vary  for  the  different  aircraft  mods.  Thus,  this  builds  in 
more  flexibility  for  the  decision  maker.  To  reflect  this 
change,  the  tree  has  been  expanded  to  include  "R&M"  decision 
elements  for  all  the  TAF  aircraft.  Thus,  there  could  be 
branches  for  the  F-4,  F-15,  F-16,  A-10,  F-ll,  E-3,  EC-135, 
and  the  EC-130.  The  weight  for  each  of  these  "R&M"  nodes  is 
(1/Number  of  TAF  aircraft  on  which  mods  are  being  done)  if 
the  mission  importance  is  the  same  for  each  aircraft  type. 
However,  if  the  mission  importance  of  each  aircraft  is 
different,  then  the  weights  may  reflect  this  fact  (see 
Figure  7).  The  only  requirement  is  that  the  weights  sum  to 
one.  Under  each  of  these  "R&M"  decision  elements  are  the 
"reliability  improvement"  and  "maintainability  improvement" 
decision  elements.  The  panel  members  will  thus  compare  K&M 
improvement  for  all  the  mods  within  a  certain  aircraft  type. 
They  must  use  the  same  scale  for  all  these  comparisons, 
however,  so  that  their  mod  values  will  be  compatible,  (see 
discussion  on  weighting  in  Chapter  5). 

Lflsal  R&M  2QQQ  Tr&S- 

Although  the  new  hierarchy  tree  proposed  in  the 
previous  section  is  an  Improvement  over  the  one  currently 
used  by  TAF,  it  is  not  the  ideal  one.  This  section  will 
suggest  a  hierarchy  tree  that  considers  all  of  the  R&M  2000 
goals.  Due  to  its  complexity,  only  some  ideas  on  the 
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structure  of  the  tree  and  the  data  it  will  require  will  be 
presented . 

First,  one  needs  to  recognize  the  goals  of  the  R&M  2000 
plan.  R&M  2000  hopes  to  increase  war  fighting  capability, 
increase  the  survivability  of  the  combat  support  structure, 
decrease  the  mobility  requirements  per  deploying  unit, 
decrease  the  manpower  requirement  per  unit  of  output,  and 
decrease  costs  (24:v).  Notice  that  the  tree  used  by  TAF 
only  directly  addresses  the  cost  issue  and  very  indirectly 
addresses  the  other  four  issues.  Thus,  the  TAF  tree  needs 
to  be  considering  much  more.  The  first  two  levels  of  the 
tree  would  be  as  shown  in  Figure  8.  The  main  goal  is  to 
satisfy  the  R&M  2000  goals.  These  five  goals  constitute  the 
second  level  of  the  hierarchy. 

Given  the  five  decision  elements  of  the  tree, 
quantitative  criteria  have  been  developed  to  measure 
progress  toward  their  achievement.  These  criteria  were 
devised  considering:  the  relevance  to  the  wartime  mission, 
conformity  to  existing  data  systems  and  reports  data, 
suitability  to  contract  requirements,  administration,  and 
warranties,  and  feasibility  considering  technology  and  cost 
(24:1-5).  These  criteria  are  depicted  in  the  following 
table . 

One  will  quickly  notice  that  reliability  and 
maintainability  are  the  most  prevalent  criteria  to  be 
considered.  However,  each  goal  is  ultimately  considering  a 
different  aspect  of  R&M.  For  example,  the  war  fighting 


Table  2.  R&M  2000  Goals  and  Criteria 


Goal  Criteria 

war  fighting  capability  reliable  ( MTBCF ) 

maintainable 

survivability  eliminate  need  for 

intermediate  support 
facilities 

mobility  reliable 

(spares,  SE,  personnel) 

manpower  reliable 

maintainable-less  personnel 

cost  reliable 

maintainable-lower  LCC 

capability  goal  is  concerned  with  the  mean  time  between 
critical  failure,  whereas  the  mobility  goal  is  concerned 
with  how  reliability  can  reduce  the  number  of  spares, 
support  equipment,  and  personnel.  Going  into  a  bit  more 
detail,  let's  consider  each  goal  separately  in  terms  of  what 
data  should  be  considered  when  comparing  the  mods. 

The  warfighting  capability  goal  may  need  to  consider 
the  percent  decrease  in  the  number  of  critical  failures, 
mean  downtime,  mission  capable  rate,  combat  reliability,  or 
availability.  The  main  question  that  needs  to  be  answered 
is:  How  can  I  capture  the  contribution  of  R&M  mods  to  combat 
capability? 

The  survivability  goal  may  need  to  consider  what  parts, 
support  equipment,  and  personnel  requirements  are  eliminated 


by  performing  the  mod.  Also  to  be  thought  about  is  the 
elimination  of  the  need  for  intermediate  support  facilities. 

The  mobility  requirement  goal  is  a  bit  more  involved 
than  the  survivability  goal.  What  is  important  is  a 
decrease  in  the  size,  weight,  or  both  size  and  weight,  of 
the  spares,  support  equipment,  and  personnel  needed.  The 
number,  size,  and  weight  of  spares  and  support  equipment 
will  need  to  be  determined.  Also,  the  decrease  in  the 
number  of  personnel  required  will  need  to  be  known.  Then, 
the  total  volume  and/or  weight  saved  by  performing  the  mod 
can  be  calculated.  To  be  considered  when  doinq  the 
calculations  is  the  scenario  you  are  dealinq  with  (i.e.  are 
you  concerned  with  the  savings  for  a  30  day  war  or  for  one 
individual  break?).  Also  a  concern  may  be  the  fact  that 
certain  spares,  support  equipment,  and  personnel  are  shared 
which  may  affect  the  calculations  or  their  meaning. 

Another  tricky  goal  to  work  with  is  the  one  to  decrease 
manpower  requirements.  If  the  break  rates  are  decreased  and 
the  repair  time  is  decreased,  obviously  the  manpower 
requirements  should  decrease.  Thus,  one  needs  to  consider 
the  mean  manhours  per  operating  hour,  the  mean  repair  time, 
and  the  combat  reliability  rate.  To  complicate  matters, 
rules  may  exist  stating  that  there  must  be  a  certain  number 
of  people  working  certain  repairs  for  safety  reasons.  What 
needs  to  be  done  is  show  how  11  these  different  pieces  of 
data  fit  together  to  yield  a  good  measure  of  the  new 
manpower  requirements  for  the  mod. 


The  measure  for  the  cost  goal  should  be  relatively 
straightforward  to  develop.  First,  the  mod  cost  and  life 
cycle  cost  savings  per  year  need  to  be  determined.  Then  the 
present  values  of  these  streams  of  expenses  and  savings  need 
to  be  calculated  using  an  agreed  upon  discount  rate.  The 
final  step  is  to  subtract  the  present  value  of  the  expenses 
from  the  present  value  of  the  savings  in  order  to  get  the 
net  present  value  of  savings. 

Even  though  most  of  the  five  decision  elements  of  this 
R&M  2000  tree  boil  down  to  one  equation  that  needs  to  be 
developed  to  compare  the  old  system  to  the  new  one,  it  is 
clear  that  the  data  needed  to  evaluate  each  mod's 
contribution  to  the  goals  is  quite  extensive.  However,  the 
task  of  data  acquisition  may  be  simplified  greatly  with  the 
use  of  a  new  model  developed  by  Synergy  (5).  This  model, 
called  SCOPE  MOD,  will  provide  data  on  the  warfighting 
capability,  manpower  requirements,  deployment  requirements, 
and  life-cycle  costs  of  the  proposed  mods  versus  their 
baseline  cases  (17).  Thus,  soon  the  ideal  R&M  2000  tree  may 
be  quite  easy  to  acquire  data  for  and  subsequently 
appropriate  to  use  in  prioritizing  the  mods. 


Problems  within  the.  AHE. 


Given  that  TAF  uses  the  AHP  correctly,  they  are  still 
In  danger  of  committing  some  errors  that  are  hidden  within 
the  AHP.  Three  of  these  AHP  drawbacks  will  be  discussed  in 
this  chapter.  The  first  one  centers  around  the  issue  of 
dependency  among  the  mods.  The  second  problem  addresses  the 
fact  that  the  AHP  cannot  deal  with  uncertainty.  The  last 
problem  occurs  in  the  comparison  step  and  involves  the  need 
for  a  consistent  comparison  scale. 

Dependency 

The  issue  of  dependency  among  the  mods  is  one  that  TAF 
is  likely  to  encounter.  Consider,  for  example,  two  mods 
that  may  be  done  on  the  radio  of  the  F-16  aircraft.  The 
first  mod  proposes  to  replace  certain  components  of  the 
current  radio  with  more  reliable  ones.  The  second  mod 
proposes  to  replace  the  entire  radio.  Obviously,  only  one 
of  these  two  mods  will  be  undertaken.  We  call  these  mods 
dependent.  It  is  acceptable  that  TAF  consider  both  of  these 
mods  when  they  perform  the  comparisons  of  all  potential 
mods.  The  prioritized  list  will  give  the  mods  in  preferred 
order,  and  the  first  mod  in  this  set  of  two  dependent  mods 
will  be  chosen.  The  reduced  prioritized  list  will  thus 
consist  of  all  the  original  independent  mods  and  the  hiqher 
valued  of  the  two  dependent  mods.  The  problem  that  may  now 
occur  is  that  if  the  AHP  is  run  on  this  reduced  set  of  mods. 
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their  order  may  be  changed  to  yield  an  Incorrect  ordering. 

An  illustration  o£  this  point  follows. 

Consider  four  mods  that  need  to  be  prioritized.  There 
are  two  dependent  mods,  mod  A  (new  radio  components)  and  mod 
D  (new  radio).  Mod  B  is  a  new  radar  and  mod  C  is  a  new 
brake  system.  Performing  the  AHP  on  these  mods  with  two 
decision  elements,  cost  and  benefit,  the  following  pairwise 
comparison  matrices  are  developed.  Note  that  "E"  means  that 
the  mod  in  the  row  is  "equal"  to  the  mod  in  the  column,  an 
"A"  signifies  that  the  rood  is  "absolutely  preferred",  and 
"1/A"  means  that  the  mod  in  the  column  is  "absolutely 
preferred"  to  the  mod  in  the  row. 


Running  these  comparison  matrices  on  Expert  Choice  (7) 
using  the  tree  that  assigns  the  weight  of  .4  to  cost  and  .6 
to  benefit,  yields  the  following  vector  of  mods  with  their 
associated  values. 


Mod 


Value 


A 

B 

C 

D 
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Assume  now  that  we  take  out  mod  D  since  It  is  dependent 
on  mod  A,  which  is  the  higher  valued  mod.  The  prioritized 
order  o£  mods  should  now  be  A,B,C.  However,  if  these  mods 
are  run  on  the  same  tree  with  the  same  comparison  matrices 
(minus  the  comparisons  done  on  mod  D)  since  "it  is 
reasonable  to  assume  that  the  decision  maker  would  still 
consider  the  criteria  to  be  of  equal  importance  (1:228)", 
the  vector  of  the  mods  with  their  associated  values  is: 

Mod  Value 

B  .527 

A  .  382 

C  .091 

Note  that  the  order  is  B,A,C  versus  A,B,C  we  would  have 
expected.  This  new  order  for  the  mods  A,  B,  and  c  is  the 
correct  order  if  only  these  three  mods  were  in  the  original 
set  of  potential  mods.  However,  mod  D  was  also  a  potential 
mod  that  was  dependent  with  mod  A  and  when  mod  D  was 
included  in  the  pairwise  comparison  matrices  a  different 
ordering  of  the  mods  A,  B,  and  C  occurred.  Thus,  since  the 
ordering  of  these  three  mods  changed  when  one  mod  was 
deleted,  obviously  there  is  a  problem  with  the  method  the 
AHP  uses  to  solve  the  pairwise  comparison  matrices. 

To  clarify  the  concern  here,  consider  the  following 
example.  Assume  you  have  an  apple,  an  orange,  a  banana,  and 
a  peach  and  you  need  to  rank  them  in  order  of  preference. 
When  you  consider  all  four  pieces  of  fruit  the  preferred 


order  is  peach,  orange,  apple,  banana.  Now,  take  away  the 
orange  as  a  potential  choice.  You  would  then  say  your 
preferred  order  is  peach,  apple,  banana.  In  this  situation, 
however,  the  AHP  may  yield  a  preference  order  of  apple, 
banana,  peach.  This  is  where  the  problem  is. 

There  are  two  ways  for  TAF  to  handle  this 
Inconsistency.  First,  TAF  may  deal  with  this  problem  by 
simply  being  aware  of  the  possibility  that  if  they  run  the 
AHP  on  the  reduced  set  of  mods,  the  orderinq  may  change.  If 
they  are  only  concerned  with  the  order  of  the  mods,  and  not 
in  the  final  value  for  the  chosen  mods,  the  easiest  solution 
would  thus  be  to  take  the  first  prioritized  list,  which 
Included  all  the  mods,  and  use  it  to  list  their  mods  in 
order  of  preference.  On  the  other  hand,  if  TAF  needs  the 
correct  values  associated  with  each  mod,  they  will  need  to 
handle  the  problem  in  a  manner  prescribed  by  Belton. 

Belton's  article  states  that  "the  only  aspect  that  has 
changed  is  the  normalization  factor  (1:229)",  so  the  only 
way  to  explain  the  reordering  (inconsistency)  is  in  the 
second  step  of  the  AHP.  It  is  in  this  step  that  the 
"vectors  which  denote  the  relative  Importance  of  options 
with  respect  to  individual  criteria  are  normalized  so  that 
their  entries  sum  to  one  (1:229)".  Saaty's  method 
normalizes  the  vectors  so  they  sum  to  one,  whereas  Belton's 
article  advocates  normalizing  the  eigenvectors  so  the 
maximum  entry  is  one.  For  example,  given  the  following 


matrix: 
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1  1/9  1 

9  19 

1  1/9  1 

Saaty's  technique  calculates  the  vector  by  a  two  step 

process.  First,  each  entry  Is  divided  by  Its  column  total. 

This  yields  the  following  normalized  matrix: 

1/11  1/11  1/11 

9/11  9/11  9/11 

1/11  1/11  1/11 

Second,  each  row  is  summed  up  and  divided  by  the  number  of 
entries  in  the  row.  This  yields  the  vector  (1/11,  9/11, 
1/11).  In  contrast,  Belton's  method  normalizes  the 
eigenvectors  of  the  original  matrix  so  that  "the  maximum 
entry  is  one  rather  than  the  entries  summing  to  one 
(1:229)".  This  technique  yields  the  following  normalized 
matrix: 


1/9 

1/9 

1/9 

1 

1 

1 

1/9 

1/9 

1/9 

Summing  each  row  of  this  normalized  matrix  and  dividing  by 
the  number  of  entries  in  the  row  yields  the  vector 
(1/9,  1,  1/9). 

Notice  that  the  two  vectors  resulting  from  Saaty's  and 
Belton's  normalizing  techniques  differ.  This  different 
vector  calculated  with  Belton's  technique  will  preserve  the 
correct  order  of  the  mods  when  the  dependent  mods  are 
deleted  from  consideration  (1:229).  What  TAF  needs  to  do  is 
change  the  P4  model  to  reflect  this  new  way  of  calculating 
the  mod  values.  They  will  then  run  the  model  on  all  the 


mods  and  determine  which  of  the  dependent  mods  is  dominated 
by  the  other  mod.  This  dominated  mod  will  then  be 
eliminated  and  the  model  will  be  run  again  so  that  the 
values  for  the  mods  that  are  still  being  considered  can  be 
determined.  The  mod  order  yielded  from  this  run  will  be  the 
same  as  the  order  yielded  by  the  first  run  with  all  the  mods 
considered,  except  of  course  the  eliminated  mod  will  not  be 
1 isted . 

Uncertainty 

"Decision  making  requires  the  study  of  uncertainty 
(10:449)".  Kamenetsky  states  that  the  AHP  is  applied  to 
"multiple-criteria  decision-making  problems  under  certainty 
(9:705)".  The  AHP  does  not  consider  any  uncertainty.  If 
there  is  uncertainty  that  can  be  ignored,  then  this  will  not 
pose  a  problem.  But,  usually  there  does  exist  some 
uncertainty  in  a  decision,  and  the  decision  made  by  TAF  is 
no  exception.  They  may  need  to  consider  the  uncertainty  in 
their  data,  (i.e.  in  their  costs  and  R&M  improvements  on  the 
mods)  and  perhaps  even  the  uncertainty  of  war.  An 
illustration  of  how  uncertainty  can  contribute  to  the  AHP 
yielding  an  incorrectly  prioritized  list  of  mods  will  be 
used  to  explain  the  problem.  First,  a  direct  application  of 
the  AHP  will  be  done  on  four  potential  mods.  Second,  a 
decision  analysis  type  application,  which  considers 
uncertainty,  will  be  done  on  the  same  four  mods.  Third,  a 


comparison  between  the  outcomes  will  be  made  to  show  AHP's 
shortcomings  in  this  area. 

AHP  Application.  The  AHP  tree  Is  a  simple  one  consisting 
of  only  two  decision  elements.  Benefit  Is  the  first  element 
with  the  weight  of  .7  and  cost  is  the  second  with  the  weiqht 
of  .3.  The  data  needed  on  the  four  mods  will  be  the 
benefit,  which  will  be  assumed  equal  for  all  four  mods,  and 
the  life  cycle  cost  savings  for  each  mod.  The  data  on  the 
cost  savings  are  found  in  the  following  table.  The  median 
value  Is  the  value  that  TAF  would  normally  use  when  making 
comparisons  between  the  mods.  The  low  and  high  values 
reflect  uncertainty  in  the  true  life  cycle  cost  savings. 

The  degree  of  the  uncertainty  may  be  a  function  of  how  much 
new  technology  needs  to  be  developed,  fuzziness  about  the 
actual  materials  needed  and  what  their  cost  will  be,  and 
uncertainty  In  how  well  the  mod  will  actually  work  .  Thus, 
a  mod  that  will  require  some  unknown  materials  and  new 
technology  will  have  a  larqe  spread  between  the  low  and  high 
costs.  If  all  goes  well  and  the  materials  required  are 
readily  available,  then  the  low  cost  may  be  achieved. 
However,  if  the  technology  does  not  develop  as  anticipated 
and  the  materials  needed  are  scarce,  the  higher  end  of  the 
cost  estimate  may  be  quickly  reached.  However,  a  mod  that 
Is  using  developed  and  proven  technology  and  knows  what 
materials  will  be  needed  will  not  have  a  large  spread 
between  the  low  and  high  cost  estimates. 


Life  Cycle  Cost  Savings  (dollars) 


low 

median 

high 

255 

260 

390 

700 

870 

880 

150 

270 

400 

650 

780 

1200 

Using  this  data  there  are  three  ways  to  use  the  data  to 
implement  the  AHP.  First,  consider  the  low  dollar  savings. 
In  this  case,  the  mod  order  (best  to  worst)  would  be 
2, 4, 1,3.  Second,  the  high  dollar  savings  could  be  used 
which  yields  the  order  of  4, 2, 3,1  for  the  mods.  Third,  the 
median  value  could  be  used.  This  provides  a  mod  list  of 
2, 4, 3,1.  Using  the  different  approaches  may  obviously  yield 
different  prioritized  lists. 

Decision  Analysis  Application.  Using  the  approach  to 
solving  a  decision  analysis  problem  as  explained  in  Xhe 
Principles  aM  Applications  al  Pec  1  Si  on  Analysis  by  Howard 
and  Matheson  (11:47-52),  the  decision  tree  is  set  up  as 
shown  below  in  Figure  9.  Note  that  there  is  a  50%  chance 
that  the  life  cycle  cost  savings  will  be  at  its  medium  value 
and  a  25%  chance  it  will  be  at  its  high  or  low  value.  Also 
note  that  the  value  function  is  determined  by  the  dollar 
savings.  The  expected  value  of  the  mods  are  given  in  the 
following  table. 


Expected  Value 

291.25 

830.00 

272.50 

852.50 
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The  following  Is  a  summary  table  of  the 


various  mod  prioritizations  yielded  by  the  two  approaches  of 
the  AHP  and  decision  analysis. 


Prioritizations  Yielded  by  the  AHP  and  DA 


Table  3. 


Approach 

AHP 

low 

high 

median 

Decision  Analysis 


Mod  Prioritization 


2  4  13 
4  2  3  1 
2  4  3  1 

4  2  13 


One  can  readily  see  that  the  rankings  are  not  very 
consistent.  The  decision  analysis  ranking  is  the  only  one 
that  considers  the  uncertainty  in  the  data  explicitly. 
Notice  that  none  of  the  AHP  prioritizations  yield  the  same 
answer.  Therefore,  when  uncertainty  is  present  TAF  may 
consider  calculating  expected  values  for  the  uncertain 
data.  These  expected  values  would  then  be  used  when  the 
panel  performs  the  pairwise  comparisons  of  the  mods. 


Seal s 


The  AHP  requires  that  the  same  "scale”  be  used  for 
decision  elements  on  the  same  level.  By  "scale"  we  mean 
the  range  used  on  the  teeter-totter  that  is  referenced  when 
making  the  pairwise  comparisons  to  determine  the  relative 
Importance  of  one  mod  versus  another,  if  the  same  scale  is 


not  used,  the  AHP  results  may  not  be  valid.  A  simple 
example  will  clarify  what  the  potential  problem  is. 


Remember  that  the  teeter-totter  we  are  referring  to  is 
symbolized  by  the  following. 


Mod  A  A  VS  S  M  E  M  S  VS  A  Mod 


In  this  diagram,  E  means  that  mod  A  is  equal  to  mod  B.  On 
the  left  side  of  the  diagram,  an  A  means  that  mod  A  is 
absolutely  preferred  to  mod  B,  a  VS  means  that  it  is  very 
strongly  preferred,  an  S  means  that  it  is  strongly 
preferred,  and  an  M  means  that  it  is  moderately  preferred. 

On  the  right  side  of  the  diagram,  the  same  definitions  hold 
but  now  mod  B  is  preferred  to  mod  A.  There  are  also 
preferences  located  between  those  shown,  for  example  a 
rating  of  M-S  means  that  a  mod  is  moderately  to  strongly 
preferred . 

In  this  discussion,  the  "large  scale"  will  refer  to  the  full 
range  of  values  (i.e.  the  whole  teeter-totter).  The  "small 
scale"  will  refer  to  only  a  half  of  the  scale  (i.e.  the 
values  between  and  including  the  "S"  rating). 

The  simple  decision  among  four  mods  that  will  be 
illustrated  has  mod  cost  and  mod  reliability  as  its  two 
decision  elements.  Both  are  weighted  equally.  Pairwise 
comparisons  using  the  "large  scale"  yield  the  following 
comparison  matrices. 


cost 

A 

B 

c 

0 

reliability 

A 

B 

C 

l) 

A 

E 

1/VS 

M 

1/M 

A 

r 

E 

VS 

S 

VS 

B 

VS 

E 

A 

S 

B 

1/VS 

E 

1/M 

E 

C 

1/M 

1/A 

E 

1/S 

C 

1/S 

M 

E 

M 

D 

M 

1/S 

S 

E 

D 

1/VS 

E 

1/M 

E 

Solving  these  matrices  using  Expert  Choice  (7),  yielded  the 
following  values  for  the  mods. 


Mod  Value 

A  .376 

B  .367 

C  .  118 

D  .139 

Thus,  the  prioritized  list  is  A,B,D,C. 

The  same  comparison  matrix  for  the  cost  was  used  in  the 
next  run  of  Expert  Choice  (7),  but  the  reliability  matrix 
was  cnanged.  This  matrix  was  adjusted  to  the  "small  scale" 
versus  the  "large  scale"  of  the  cost  matrix.  For  example, 
if  mod  A  was  rated  "S"  versus  mod  C,  the  new  matrix  changes 
the  rating  to  a  "M" .  The  following  conversion  table  was 
used  to  make  all  the  changes  in  the  reliability  matrix. 

Large  Scale  =  Small  Scale 

E  E 

M  E-M 

S  M 

VS  M-S 

A  S 

The  resulting  comparison  matrix  for  reliability  is  shown 
below. 


reliability  A  B  C  D 

i - 


Solving  for  the  mods  using  the  "large  scale"  for  the  cost 
and  the  "small  scale"  for  the  reliability  yielded  the 
following  values. 

Mod  Value 

A  .316 

B  .  389 

C  .133 

D  .162 

Thus,  the  prioritized  list  is  now  B,A,D,C. 

Notice  that  when  the  reliability  matrix  was  chanqed 
from  the  large  scale  to  the  small  scale,  the  mods'  priority 
changed.  The  reason  for  this  inconsistency  is  alluded  to  by 
Watson.  His  article  on  assessing  attribute  weights  states 
that  "if  a  change  of  scale  is  made  on  one  attribute  of  a 
multiattribute  value  function,  then  the  weight  must  change 
on  that  attribute,  to  preserve  the  order  induced  by  the 
value  function  (26:583)".  In  our  illustration,  the  value 
function  is  represented  by: 

.5(cost)  +  . 5 ( rel iabi 1 i ty ) 

When  the  scale  on  reliability  was  changed,  we  would  have 
needed  to  change  the  coefficient  for  that  attribute  in  the 
value  function  to  preserve  the  mod  order. 

For  example,  when  the  large  scale  was  used  on  both 
comparison  matrices  the  mod  order  w«>s  A,B,D,c.  However, 
when  the  scale  on  the  reliability  branch  was  chanqed  to  the 
small  scale,  the  mod  order  was  B, A, D  C .  To  convert  the 
B, A, D, C  order  back  to  the  A,B,D,C  order  (the  correct  order 


since  it  was  obtained  when  the  same  scales  were  used  to  make 
the  comparisons)  we  would  need  to  change  the  coefficients  of 
cost  and  reliability  in  the  value  function.  Thus,  perhaps  a 
weight  of  .4  for  cost  and  .6  for  reliability  would  convert 
the  B,A,D,C  order  to  A,B,D,C. 

This  changing  of  the  weights  can  get  very  confusing, 
but  luckily  it  is  not  necessary  for  TAF  to  do  this.  The 
only  thing  they  need  to  do  is  remember  to  use  the  same  scale 
(i.e.  the  large  scale  that  ranges  from  ME"  to  "A")  when 
doing  the  mod  comparisons.  The  easiest  way  to  accomplish 
this  is  by  considering  the  best  and  the  worst  mod  for  each 
particular  decision  element.  The  value  of  the  worst  mod  is 
subtracted  from  the  value  for  the  best  mod.  This  difference 
is  then  divided  by  four  (the  number  of  increments  on  the 
teeter-totter).  This  quotient  is  the  amount  that  each 
Increment  on  the  scale  stands  for.  Thus,  if  the  quotient  is 
100  and  mod  A  has  a  value  of  200  and  mod  B  has  a  value  of 
100,  then  mod  A  will  be  rated  as  moderately  better  (one 
Increment)  than  mod  B.  Thus,  this  method  provides  an  easy 
way  to  compare  the  mods  and  also  assures  that  the  best  mod 
when  compared  to  the  worst  mod  for  any  specific  decision 
element  will  receive  a  rating  of  MA".  This  forces  the  large 
scale  to  always  be  used  (since  the  "A"  value  is  guaranteed 
to  appear  in  every  comparison  matrix),  and  the  requirement 
is  thus  satisfied. 

It  should  be  pointed  out  that  the  tendency  to  use  a 
small  scale  will  usually  arise  when  comparing  data  that  is 


very  close.  For  example,  if  the  life  cycle  cost  savings  for 
four  mods  are  10,  11,  12,  and  13,  then  the  comparison  matrix 
might  be  similar  to  the  following  since  the  values  are  not 
vastly  different  from  one  another. 


LCC 

1 

2 

3 

4 

1 

E 

1/E-M 

1/M 

1/M-S 

2 

E-M 

E 

1/E-M 

1/M 

3 

M 

E-M 

E 

1/E-M 

4 

M-S 

M 

E-M 

E 

The  matrix  may  even  look  like  the  following  to  reflect  no 
difference  between  the  mods  with  respect  to  life  cycle  cost 
savings . 


LCC  12  3  4 


1 

2 

3 

4 


E 

E 

E 

E 


E  E  E 
E  E  E 
E  E  E 
E  E  E 


However,  if  the  large  scale  is  used,  the  following  matrix 
would  be  developed. 


LCC  12  3  4 


1 

2 

3 

4 


E  1/M  1/S  1/VS 

M  E  1/M  1/S 

S  M  E  1/M 

VS  S  M  E 


Notice  that  the  matrices  are  different,  and  as  discussed 


above,  this  may  cause  inconsistency  in  the  final  values  for 


the  roods . 


This  chapter  discusses  new  techniques  that  TAF  might 
consider  using  to  prioritize  their  potential  mods.  The 
techniques  of  goal  programming,  a  proxy  approach  to  multi¬ 
attribute  decision  making.  Data  Envelopment  Analysis,  and  a 
multiattribute  decision  making  by  sequential  resource 
allocation,  were  all  considered  as  potential  methods  for  TAF 
to  use  in  their  mod  prioritization  problem.  However,  none 
of  these  techniques  were  found  to  be  suitable.  Goal 
programming  is  not  set  up  to  solve  TAF ' s  type  of  decision 
problem.  For  example,  it  will  indicate  how  many  of  each  mod 
to  do,  given  certain  constraints,  but  this  is  not  what  TAF 
needs  to  know  (i.e.  TAF  needs  to  rank  the  mods)  (14:518). 

The  proxy  approach  is  "practical  for  decision  making  under 
certainty  (15:687)",  thus  it  would  not  help  TAF  to  deal  with 
their  uncertainty.  The  Data  Envelopment  Analysis  is  an 
"efficiency  measurement  methodology  (3:2)",  and  is  not  at 
all  applicable  to  TAF's  situation.  The  sequential  resource 
allocation,  like  goal  programming,  is  in  a  format  that  is 
not  suitable  for  use  by  TAF  (13). 

However,  two  techniques  were  found  that  are  suitable 
for  use  by  TAF.  The  techniques  of  decision  analysis  (DA) 
and  Preference  Ranking  Organization  Method  for  Enrichment 
Evaluations  (PROMETHEE)  will  be  explained.  First,  the 
methodology  of  each  technique  is  presented.  This  is 
followed  by  an  example  of  how  TAF  might  apply  the  technique 


to  their  specific  problem  of  prioritizing  potential  mods. 
Last,  a  brief  discussion  on  the  improvement  that  can  be 
realized  by  using  the  technique  versus  the  AHP  will  be 
given. 

Decision  Analysis 

DA  is  a  "procedure  that  can  be  applied  to  any  decision 
susceptible  to  logical  analysis  (11:27)".  It  is  Iterative 
and  comprised  of  three  phases.  The  first  phase  is  the 
deterministic  phase.  In  this  step  "the  variables  affecting 
the  decision  are  defined  and  related,  values  are  assigned, 
and  the  Importance  of  the  variables  is  measured  without  any 
consideration  of  uncertainty  (11:26)".  In  the  second  or 
probabilistic  phase,  probability  assignments  on  the 
variables  are  made  and  risk  preference  is  introduced.  "The 
third,  or  Informational,  phase  reviews  the  results  of  the 
last  two  phases  to  determine  the  economic  value  of 
eliminating  uncertainty  in  each  of  the  important  variables 
in  the  problem  (11:26)". 

Using  these  steps  an  approach  for  solving  TAF's 
prioritization  problem  is  described  below.  The  problem  has 
been  simplified  to  consider  only  three  mods.  The  first  step 
is  to  build  the  model  (see  Figure  10)  that  is  used  to  obtain 
an  expected  net  present  value  for  each  mod.  These  values 
will  then  be  ordered  high  to  low  to  prioritize  the  mods. 

The  decision  diagram  shows  the  relationship  between  the 


since  the  uncertainty  In  the  data  will  be  considered.  In 
the  diagram,  the  variables  that  are  certain  are  enclosed  by 
a  double  circle,  whereas  the  uncertain  variables  are 
enclosed  by  a  single  circle.  The  only  uncertainty  in  this 
example  concerns  the  life  cycle  cost  (LCC)  savings  per  mod 
per  year . 

Note  that  the  decision  diagram  depicted  in  Figure  10  is 
an  influence  diagram.  Software  exists  to  solve  for  the 
expected  net  present  value  of  models  constructed  in  this 
manner . 

The  second  step  is  to  assign  probabilities  to  the 
uncertain  variable.  The  probabilities  will  normally  be 
determined  by  consultation  with  experts  in  the  related 
areas.  Arbitrary  probabilities  are  used  in  this  example. 
Their  values  appear  below. 

Probabilities 


LCC  Savings 

High  .25 

Median  .50 

Low  .25 

The  third  step  is  to  calculate  the  expected  net  present 
value  of  each  mod  using  the  mod  value  function.  in  this 
example,  the  value  for  each  mod  is  calculated  using  the 
following  relation. 

Value  of  a  mod  *  mod  savings  * 

total  LCC  savings  -  total  mod  cost® 

(LCC  savlngs/mod/year )*( Inventory  size)*(#  years  left)  - 
(mod  cost/a lrcraft )*( inventory  size) 


This  relation  is  depicted  in  the  decision  diagram. 


The  following  data  Is  used  for  this  example. 
LCC/mod/year 


Mod 

high 

median 

low 

mod  cost/AC 

inventory  size 

#  years 

A 

40 

35 

20 

40 

100 

20 

B 

30 

25 

20 

30 

150 

20 

C 

60 

40 

10 

50 

100 

10 

To  solve  for  the  value  of  each  mod,  the  uncertainty  in 
the  LCC  savings/mod/year  will  be  accounted  for  by  using  the 
following  equation. 

Value  of  a  mod  = 

l . 25(LCCL*inv  size*#  years)  +  .5(LCCM*inv  size*#  years)  + 
.25(LCCH*inv  size*#  years))  -  (mod  cost/AC*inv  size) 

where  LCCL  is  the  low  LCC  savings,  LCCM  is  the  median  LCC 

savings,  and  LCCH  is  the  high  LCC  savings.  Note  that  in 

these  calculations  the  simplification  of  assuming  a  zero 

discount  rate  has  been  made.  This  eliminates  the  need  to 

use  a  more  complicated  equation  to  calculate  the  mod  value 

(see  the  discussion  in  Chapter  4  in  the  section  "improvement 

to  TAF's  Tree").  Also,  note  that  software  exists  to  find 

the  expected  net  present  value  for  each  mod  given  the  model 

in  an  influence  diagram  form  as  in  Figure  10.  The 

calculated  values  for  the  mods  A,  B,  and  C  appear  below. 

Mod  Value 

A  61,000 

B  70,500 

C  32,500 

Therefore,  the  mod  order  is  B,A,C. 

There  are  two  reasons  TAF  may  want  to  consider  using 
DA.  First,  DA  is  able  to  "represent  the  uncertainty  that 
Inevitably  permeates  a  decision  problem  (11:24)".  This 


paper  (Chapter  5)  already  made  the  point  that  the  AHP  does 
not  deal  with  uncertainty,  which  is  one  of  its  major 
shortcomings.  The  decision  problem  TAF  faces  does  contain 
uncertainty  in  the  data,  and  DA  will  handle  this  uncertainty 
correctly.  Second,  by  putting  all  the  values  (i.e.  both  the 
mod  savings  and  the  mod  mission  benefit)  in  terms  of  dollars 
a  firm  foundation  for  analysis  is  provided.  It  is 
straightforward  to  compare  mods  when  their  values  are 
reduced  to  one  common  measure  (i.e.  dollars)  that  is 
understandable  to  all.  The  decision  process  as  explained 
above  still  requires  some  work,  especially  in  the  area  of 
developing  a  way  to  express  the  mission  benefit  in  terms  of 
dollars  so  the  expected  net  present  value  of  the  mod  will 
also  be  in  dollars.  However,  once  this  task  is  successfully 
accomplished,  TAF  would  be  able  to  use  DA  to  prioritize 
their  mods. 

PBQMETHEE 

The  PROMETHEE  is  a  "new  class  of  outranking  method  in 
multicriteria  analysis  (4:228)"  that  is  used  to  select  and 
rank  projects.  The  new  method  was  introduced  by  Roy  and 
Includes  two  phases.  The  first  phase  constructs  an 
outranking  relation  on  a  finite  set  of  actions.  This 
Involves  defining  a  preference  index  for  the  decision  maker 
which  determines  the  outranking  relation.  The  second  phase 
exploits  the  outranking  relation  in  order  to  yield  an  answer 
that  "maximizes  the  multicriteria  problem 


Max  (£l(a), . . . ,fk(a)  ja  an  element  of  the  set  K>  where  K  Is 


a  finite  set  of  actions  and  fi  1=1,... ,k,  are  k  criteria  to 
be  maximized  (4:228)". 

An  illustration  of  an  example  application  will  help  to 
explain  the  PROMETHEE.  Consider  the  multicriteria  problem 
of  ranking  three  mods  (A,B,C).  Three  criteria  are 
considered  relevant  to  the  decision  maker:  fl:reliability 
improvement,  f 2 :malntainabillty  improvement,  and  f3:LCC 
savings.  For  each  criterion,  a  criterion  type  (generalized 
criteria)  indicating  the  preference  function  of  the  decision 
maker  must  be  specified.  There  exist  six  types  of 
generalized  criteria  (4:229-232),  five  of  which  have 
parameters  that  are  specified  by  the  decision  maker.  For 
our  example  we  will  assume  all  the  criteria  have  the  same 
generalized  criteria,  which  will  be  the  "usual  criterion". 
The  usual  criterion  assigns  the  value  of  one  to  a  comparison 
between  two  mods  if  there  exists  a  difference  between  them, 
and  a  zero  if  no  difference  exists.  Thus,  "there  is 
indifference  between  A  and  B  if  and  only  if  the  value  of  A 
equals  the  value  of  B.  As  soon  as  the  two  evaluations  are 
different  the  decision  maker  has  a  strict  preference  for  the 
action  having  the  greatest  evaluation  (4:229)". 

The  data  gathered  on  the  three  mods  is  shown  below. 


Mod 


LCC  savings 


RI 


MI 


where  RI  stands  for  reliability  Improvement  and  MI  stands 
for  maintainability  improvement.  Using  the  data  and 
considering  the  preference  function  explained  above,  the 
calculations  on  the  preference  index  are  made.  The 
preference  index  is: 

*  k 

n  (a,b)  =  (  ^  nt  P'  (a,b))/(^  n;) 
i*  ( 

where  FI;  is  a  weight  denoting  the  relative  importance  of 
criterion  fi.  In  this  example,  n,  *.35,  Tl*  *.35,  and 
Il3  =.3.  Using  this  information,  the  following  table  is 
developed . 


ABC  sum  of  row 


A 

- 

.65 

.65 

1.30 

B 

0 

- 

.35 

.  35 

C 

0 

.65 

- 

.65 

sum  of  column 

0 

1.30 

1.00 

For  example,  the  entry  for  A  versus  B  is  calculated  by 
I ( . 35 ) * ( 0 )  +  ( . 35 ) * ( 1 )  +  ( . 3 ) * ( 1 ) 1  /  l ( . 35 )  +  (.35)  ♦ 

( .3) 1-.65 

The  sum  of  the  column  is  now  subtracted  from  the  sum  of  the 
row  for  each  mod,  yielding  the  values: 


mod 

A  * 

(1.3)- 

-(0)  = 

1.3 

mod 

B  - 

(  .35) - 

(1.3) 

*  -.95 

mod 

C  - 

(  .65)- 

(1)  - 

-.35 

Thus,  the  ranking  of  the  mods  (high  mod  value  to  low  mod 
value)  is  A,  C,  B. 


The  advantage  of  using  this  method  as  compared  to  using 
the  AHP  is  that  it  may  avoid  the  sensitivity  to  small 
changes  that  is  experienced  with  the  AHP.  This  problem  (as 
discussed  in  Chapter  5)  may  cause  the  ranking  to  change  when 
there  are  small  changes  made  in  the  comparisons. 
Specifically,  if  a  small  scale  is  used,  one  ranking  will  be 
obtained  and  if  the  large  scale  is  used,  a  different  ranking 
may  be  obtained.  In  the  PROMETHEE,  "some  small  deviations 
in  the  determination  of  these  values  (the  parameters  for  the 
different  preference  functions]  do  not  often  induce 
Important  modifications  of  the  obtained  rankings  (4:228)". 
Since  the  decision  maker  cannot  usually  fix  the  exact  values 
for  the  parameters,  this  "stability  problem  is  of  a  major 
importance  (4:235)".  However,  the  parameters  should  have  an 
economic  significance  which  should  help  the  decision  maker 
to  fix  them  more  easily  and  precisely  (4:236).  TAF  may 
decide  to  Implement  the  PROMETHEE  since  it  is  simple,  clear, 
and  stable.  Further  work  would  need  to  first  be  done  on 


determining  the  correct  preference  functions  and  parameters 
for  each  of  the  criterion  functions,  but  once  this  is 
accomplished,  the  rest  would  be  straightforward. 


Val idatlon .  This  thesis  presents  a  three  part 
validation  of  the  decision  making  process  TAF  uses  to 
prioritize  their  R&M  mods.  The  first  part  of  the  validation 
focuses  on  the  general  steps  TAF  must  perform  and  three 
specific  criteria  TAF  must  meet  to  ensure  their  final 
decision  made  is  a  good  decision.  TAF  does  not  satisfy 
these  requirements  in  two  areas.  First,  the  data  from  MODAS 
needs  to  be  more  fully  understood.  Second,  the  value 
function  for  the  mods  needs  to  be  improved  upon  so  that  it 
always  assigns  a  higher  value  to  the  mod  that  TAF  prefers. 
However,  these  areas  of  deficiency  may  be  easily  corrected. 
The  applicability  of  the  AHP  to  TAF's  specific  decision 
problem  is  addressed  and  the  analysis  indicates  that  the 
problem  is  ideally  suited  for  an  application  of  the  ahp 
The  second  part  of  the  validation  accomplishes  ar 
depth  analysis  of  TAF's  application  of  the  AHP.  The 
structure  of  the  hierarchy  tree  is  shewn  conr  a  •  *> 

problems .  Two  of  the  decision  element.  ' 
and  "remaining  life  of  the  system" >  bet n  * t  •  e  ••• 
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decision  element.  A  new  tree  is  presented  that  restructures 


TAF's  tree  by  combining  these  three  decision  elements  into 
one.  The  current  TAF  tree  also  fails  to  consider  all  five 
of  the  R&M  2000  goals.  Another  tree  is  presented  that 
incorporates  these  goals,  and  ideas  on  the  data  necessary  to 
evaluate  mods  using  this  tree  are  discussed. 

The  third  part  of  the  validation  presents  three  areas 
contained  within  the  AHP  that  may  pose  problems  for  TAF. 

The  area  of  dependency  between  mods  is  discussed  first. 

This  area  will  actually  pose  no  problem  as  long  as  TAF  is 
aware  of  its  existence.  A  new  method  to  deal  with  the 
dependency  is  presented.  This  method  may  easily  be 
incorporated  into  the  P4  model  and  TAF  will  eliminate  this 
potential  problem.  The  second  problem  considered  is  the 
fact  that  the  AHP  does  not  consider  uncertainty.  An  example 
is  presented  that  compares  the  results  obtained  from  using 
the  AHP  and  those  obtained  from  using  DA  (a  method  that  does 
consider  uncertainty).  The  end  result  is  that  TAF  may  want 
to  consider  uncertainty  in  their  data  by  using  DA  to  obtain 
some  values  (for  example  LCC  savings)  for  the  mods.  These 
values  would  then  be  used  when  making  the  comparisons.  The 
third  problem  discussed  concerns  the  inconsistency  in  mod 
values  that  may  be  caused  by  using  different  comparison 
scales . 

Alternative  Methods .  Two  alternative  methods  are 
presented  which  may  be  suitable  for  TAF  to  use  for  their 
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decision  problem.  The  first  method,  DA,  is  able  to 
represent  the  uncertainty  that  TAF  may  encounter  and  also 
provides  a  firm  foundation  for  the  analysis.  The  second 
method,  the  PROMETHEE,  is  a  method  that  is  used  to  select 
and  rank  projects.  The  advantage  of  this  method  is  that  it 
is  not  overly  sensitive  to  small  changes  in  the  comparison 
scale  (which  is  a  drawback  of  the  AHP ) . 

Eurther  Research 

The  areas  for  further  research  may  be  divided  into  two 
main  areas.  First,  the  data  used  to  make  the  pairwise 
comparisons  requires  more  work  to  be  done.  Second,  if  TAF 
decides  to  Implement  either  DA  or  the  PROMETHEE  method  to 
solve  their  problem,  more  work  will  needed  in  those  areas. 

Data .  A  new  tree  was  developed  for  TAF  (l.e.  NEWTREE) 
that  requires  data  on  the  LCC  savings.  This  is  not  a  simple 
area  to  work  in  since  LCC  embodies  such  an  extensive  set  of 
costs,  for  example  spares,  technical  orders,  fuel,  and 
training.  TAF  needs  to  decide  what  level  of  detail  they 
need  and  then  acquire  and  synthesize  the  relevant  pieces  of 
data . 

The  R4M  tree  also  poses  new  data  requirements  for  TAF. 
The  types  of  data  that  will  be  used  to  evaluate  each  goal, 
as  well  as  the  sources  for  some  of  this  data  must  be  decided 
upon.  This  task  may  be  simplified  by  new  software,  such  as 


SCOPE  and  MARGI ,  that  calculate  combat  capability  and  LCC, 
but  the  task  is  still  quite  involved. 

New  Methods .  If  TAF  decides  to  use  DA  to  solve  their 
prioritization  problem,  they  will  need  to  do  two  thinqs. 
First,  they  will  need  to  figure  out  a  methodology  for 
obtaining  the  relevant  probability  distributions  of  the  cost 
and  benefit  data.  Second,  they  will  need  to  figure  out  a 
methodology  for  assessing  values  of  the  alternatives  (i.e. 
what  weights  are  correct  and  what  decision  elements  are 
needed).  The  difficult  part  will  be  determining  a  value 
function  for  the  R&M  improvement  in  terms  of  dollars. 

If  TAF  decides  to  use  the  PROMETHEE  method,  they  will 
need  to  define  their  preference  functions  for  each  decision 
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